The beating heart symbolizes the guilt the narrator feels for having killed the old man.He mentions at the beginning of his narration that he is a madman.He describes how an .."evil eye.." he saw on the old man he lived with, affected him much.He decided to put an end to it but he had to kill the innocent holder of the evil eye; the old man.He feels the burden of his own guilt in the beating heart.