The government only limits our rights in particular scenarios. These primarily include instances in which exercising the right causes harm to others. It also can include instances where the right is almost impossible to protect, like our right to privacy when in public spaces