Final answer:
The Engel v. Vitale decision found school prayer unconstitutional based on the Establishment Clause of the First Amendment, as it effectively constituted state approval of religion, violating the principle of separation of church and state.
Step-by-step explanation:
The majority of the Supreme Court justices found school prayer unconstitutional in the Engel v. Vitale case primarily based on the interpretation of the Establishment Clause of the First Amendment of the U.S. Constitution. This clause prohibits the government from making any law 'respecting an establishment of religion'. The prayer, even though nondenominational and voluntary, represented an official approval of religion by the New York State, which went against the principles of the Establishment Clause. The Court's ruling was aimed at preventing the government from participating in the religious activities, thus upholding the separation of church and state. This decision set a precedent for future cases involving religious activities in public institutions, such as schools, and extended to federal, state, and local governments. Despite its controversy and the fact that such a stance is unpopular with many Americans, the Court sought to maintain a balance between maintaining liberty and avoiding the establishment of religion by the state.