In the very first sentence, it states, ''Victor and Norman are cow-owners...'' Just by this, we know that Victor hard part of the ownership over the office building.
When Norman died, this left Victor being the only person who had ownership of the office, which means he is fully in charge of it. So, Victor owns the building because of the ownership he had previous to Norman's death.