Specific heat is the amount of energy needed to increase the temperature per unit mass in 1°C.
The nature of the elements or compounds (metalic elements in this case) determines the transference of heat inside the body of the object. The mechanism of heat transfer inside the metals is conductivty. The more compact and denser the metal, the closer the atoms and the faster the transference of heat by conductivity and the lower the specific heat.
Then, the reason why specific heat of Al is much lower than gold, is that gold is much denser than Al.