I would say that the phrase only applied to white people mainly because blacks were not considered citizens and were mainly considered property. The phrase had good intentions but it took another 100 or 200 years for blacks and other minorities to gain equal rights.