It's because of how smart objects work. A smart object is just a container that holds something inside it. In this case, it's holding our image. When we scale a smart object to make it bigger or smaller, it's the size of the container that we're changing, not what's inside it.