Hey there!
Madison was an Anti-Federalists and he said that the greatest danger to the rights of an individual came from none other than the government. He thought that the government can take away rights from individuals and believed that this way it may become like the tyranny the colonies fought so hard against.
Hope this helps!