Hello. You did not enter the text to which this question refers. However, I believe that you are referring to "Heart of a Samurai" whose protagonist is Manjiro Nakatama, one of the first Japanese to immigrate to the USA.
While reading the book, we can see that the evidence that Manjiro has to believe that men are barbarians are the stories that fishermen told on the island where he lived in Japan. Fishermen claimed that barbaric men had hair on their faces and had eyes of all colors. As the fishermen were wise men who had vast knowledge, Manjiro believed what they said.