They knew he was a wizard and were worried that they would hurt or kill him like Voldemort killed his parents.
Bonus answer- They disliked him because of his close resemblance to his father.
i hope this helped you!
5.2m questions
6.7m answers