If they did in any amount view people in Africa and Asia as potential "customers," that would still represent an opportunity for investment for them. And honestly, the bankers were more concerned about revenue from the businesses participating in imperialism than they were thinking about native populations.
There certainly were plenty of bankers and merchants who held Social Darwinist beliefs, but that can't be said as a blanket statement for all.
Bankers are always in the business of making money. They are looking for investments that will produce a sizable return on their investment.