From a moral standpoint, it is wrong for one country to invade another country and seize part or all of its lands. Until relatively recently, this is, the early twentieth century, the citizens or subjects of an economically and militarily strong country expected their government to make use of those resources in expeditions aimed at putting another weaker country under partial or total control. Also, citizens or subjects enlisting for war took pride in their military "adventure" and made it appear as "service for their country" or "patriotism."
Political thinking evolved during the late nineteenth century and early twentieth century, and as world wars churned out appalling numbers of casualties and devastation across the globe, which made more and more people question the previous notions on military projection in view of the staggering price paid by belligerent nations in an industrial era. A different notion of the value of human life and peaceful co-existence emerged around the second half of the twentieth century as a result of the Cold War and the possibility of overall destruction of mankind by nuclear weapons.
A clear notion of the sovereignty (right each country has to make decisions) and self-determination (right each country has to make its own decisions) took hold and became equated to the Declaration of the Rights of Man and Citizen created during the French Revolution and refined over the years since then.
As of the territories the U.S. took over by the force of arms in the nineteenth century, it is important to bear in mind that the events leading to the annexation of those lands took place in a different time within a society with markedly different values of all kinds, so that their actions and thoughts cannot be judged by making comparisons and contrasts with our values today. Intending for the United States to give back all the lands it acquired throughout the nineteenth century and returning to its original thirteen-colony configuration not only is impossible, but undesirable: millions of people living in those lands have been born, grown and lived as U.S. citizens, so breaking up the country into its former territorial selves would create all sorts of social, political, economic, etc. chaos.