Final answer:
The formation of layers in protoplanets due to the release of particles and energy from unstable elements is caused by radioactive decay. This process creates heat, driving planetary differentiation and resulting in denser materials moving towards the core and lighter materials rising to the surface.
Step-by-step explanation:
The process that contributes to the formation of layers in protoplanets when unstable elements release particles and energy is radioactive decay. This process generates heat from the decay of radioactive isotopes, leading to what is known as planetary differentiation. During this process, denser materials, such as metals, tend to sink towards the core of the protoplanet due to gravity, whereas lighter materials, such as silicates, rise to the surface.
Activities such as fusion reactions, convection currents, and gravitational collapse all play roles in various stages of stellar and planetary formation. However, for the specific phenomenon of generating interior heat within protoplanets, leading to differentiation and layer formation, it is the heat produced by radioactive decay that is the key contributor.