Caves are formed by the limestone dissolving. Rainwater picks up carbon dioxide from the air and as it goes through the soil, which turns into a weak acid. This slowly dissolves out the limestone along the joints, bedding planes and fractures, some of which become enlarged enough to form caves.