I believe the answer is: The number of confederates increased
Confederates in this context refers to the number of accomplices that supported the wrong answer.
As the number of people increased, we started to make an assessment that make us doubt our own way of thinking. This would tend to make us choose to avoid the humiliation of being false and choose to conform instead.