African Americans didn't request the privilege to vote at that time did not fully understand their rights and did not make any moves or even campaign to be allowed that right as citizens of the united states of America.
Also, The number of African Americans were more than the White people, if they really vote, and their elected representatives win most senate or other such positions the white people will have less impact in the affairs of the government.