I believe the correct answer is that he loves his family too much to do so.
He is in love with his wife, Penelope, and he loves his son as well, so he doesn't want to do what Calypso wants him to do. He wants to honor his family and keep them proud of him, and not disappoint them by listening to Calypso.