Answer:
Annexation was never a very popular choice. Many Canadians were loyal to the Crown and Great Britain, especially the descendants of the United Empire Loyalists. French Canadians worried about being an even smaller minority in a larger union, and were concerned about American anti-Catholicism.