The main reason the Supreme Court ruled in favor of the civil rights activists by using the Constitution as their reasoning. They concluded that both the 13th and 14th amendment didn't give Congress the right and/or power to regulate affairs of individuals and/or businesses.
Hope this helps! :)