There might not be much room to live, so there can't be many people there.
There are only enough resources for a few people
There might be fighting, and people moved away
The place might be hard to reach, and not everyone wants to try.
In rare cases, the place might be restricted to only certain people.