They knew he was a wizard and were worried that they would hurt or kill him like Voldemort killed his parents.
Bonus answer- They disliked him because of his close resemblance to his father.
i hope this helped you!
5.6m questions
7.3m answers