In Gulliver's Travels, physical size is indeed a primary determinant of power, as seen in the encounters between Gulliver and the Lilliputians and then the Brobdingnagians. However, Swift also highlights other factors that enable one group to have power over another.
In the case of the Lilliputians, their power comes from their unity and organization. Despite being small in stature, they are able to control Gulliver through their military and political might. On the other hand, the Brobdingnagians are physically larger than Gulliver but also possess a moral superiority over him. They view Gulliver as a curious specimen rather than a threat and are able to dominate him through their intellect and ethical standards.
Swift is trying to illustrate that power is not just determined by physical size, but also by other factors such as unity, organization, intellect, and moral standards. He is suggesting that those who possess these traits can gain power over those who do not, regardless of physical size. Additionally, Swift is criticizing the arbitrary nature of power and the ways in which it is often abused by those who possess it.