The answer is:
Captian Anthony who was probably his father. Anthony is the clerk for Colonel Lloyd, managing Lloyd's surrounding plantations and the overseers of those plantations. Anthony is a cruel man who takes pleasure in whipping his slaves, especially Douglass's Aunt Hester.