Japan is not a single land mass but a collection of Islands.
Step-by-step explanation:
Japan has a difficult topography to manage for a nation and a centralized rule is nearly impossible because of this terrain.
It is made out of thousands of small and big islands and many of the big islands have huge mountain ranges and volcanoes on them.
What this entails is that there is a major issue of connectivity in all parts of the country and there is little way of how a centralize power could exercise it effectively.
So they have smaller local governments too.