Answer:
Toronto and Brampton largely sit on Georgian Bay Shale, which runs roughly parallel to the Niagara Escarpment all the way from Lake Ontario to the Collingwood area. (the grey area on the map). This rock is much more erosion resistant and harder than the limestone in Eastern Ontario.
And I'm from Brampton too!