Well there's a couple reasons why. First, the land itself can become “tired" and less fertile. This is because the same type of crop planted repeatedly in the same area keeps draining the land of the same nutrients needed for that plant's growth. Pest can also plant their nest near the crop if the farmer continues to plant the same crop in the same place.
Which is the reason why things such as "crop rotation" exist to keep things fresh, and prevent what I described above from happening.