Firmly fixed or attached = I caught a tremendous fish and held him beside the boat half out of water, with my hook fast in a corner of his mouth.
The hook needs to hold on to the fish so that the fisherman can pull him out of the water. Since the hook is being described as fast to the fish's mouth, that tells us it is attached.
compressed or crumpled = A green line, frayed at the end where he broke it, two heavier lines, and a fine black thread still crimped from the strain and snap when it broke and he got away.
Crimped means to be compressed or crumpled. A crimping iron used on a person's hair makes the hair have ridges like ^^^^^^^ whereas a straightening iron makes a person's hair straight, and a curling iron gives it curls.
sulky or angry = I admired his sullen face, the mechanism of his jaw, and then I saw that from his lower lip —if you could call it a lip grim, wet, and weapon like, hung five old pieces of fish-line,
The face is the one of the fish who has been caught. It's obvious he has been hooked before since there are five old pieces of fish-line hanging out of his mouth. This is most likely the first time he's been reeled in which makes him angry.