They knew he was a wizard and were worried that they would hurt or kill him like Voldemort killed his parents.
Bonus answer- They disliked him because of his close resemblance to his father.
i hope this helped you!
8.8m questions
11.4m answers