It is due to a low entropy level in the rock, and also the type of chemical bonds the rock has.
An example similar to this one would be diamond. It is made up of carbon bonds. But once you take a (real) diamond out of the ground, it very slowly, not visible to the human eye, yet surely is turning back into graphite. (another substance made by carbon bonds)
i hope this helped.