The correct way to say would be "I can't figure this out" and "I don't figure this out".
"I can't figure this out" means you lack the ability or faith in yourself to figure "this" out. You can't do it.
"I don't figure this out" sounds weird. It's possible to paraphrase it in the past or future tenses, but in the present it's usually said in present continuous, i.e "I'm not figuring this out" (as "to figure out", per se, is a process of sorts) and literally means you do not occupy yourself with figuring "this" out at this moment or generally.