Final answer:
The term used in system theory to describe the process of a system becoming more complex and specialized is Differentiation. It's a key aspect of modernization theory, where societies advance through role specialization and complex social structures.
Step-by-step explanation:
In the context of system theory, the term that describes the process of a system becoming more complex and specialized over time is b) Differentiation. Differentiation is a process often associated with the modernization of societies. It refers to the increasing complexity of the social structure through the specialization of roles and institutions. For example, as societies advance, they tend to develop distinct sectors such as industrial, educational, and healthcare, each becoming more specialized in function. This is a key aspect of modernization theory, which suggests that societies evolve toward more advanced technology and complex industry as cultural values shift to support hard work and success.