Answer:
In a word, erosion. The action of the weather on exposed rock surfaces exposes them to water in its various states (solid ice, liquid water, and if you deep enough underground where the temperature increases, steam) and the transition of water from one state to another causes it to expand (solid ice) and contract (liquid water.) So when water gets into a rock's crevices, the climate changes can slowly begin splitting it apart. On the surface, the water can begin to wear away the rock face and expose another layer, which in turn may be worn away. Where there is not a lot of weather variation, there's less erosion, and therefore older rocks. An extreme case are the rocks on the moon--with no weather, there's no erosion, and no wearing away--so moon rocks are nearly as they were when the moon was formed.