Political parties may lead to corruption and over-concentration of power in a few party leaders. Those are two of the reasons why some people were opposed to the formation of political parties.
George Washington lived and governed in a very different time. He was also heavily influenced by his experience under British rules. The Brits at the time (and even now) were ruled by political parties. So while I do not agree with Washington, his view is understandable.