Answer:
While different materials may have an impact on how the size of the shirt changes with the time, the elements that have an effect on the shirt since the beginning are especially the details in it.
For example, a bigger shirt needs more buttons than a smaller one, or bigger shirts may need a bigger design than a smaller shirt.
Another thing that affects it is the average of the population, for example, you can see that a medium in a given country is smaller or bigger than a medium in another country.