In diamond, The arrangement of carbon atoms follows a tetrahedral shape. Each carbon is attached to four other carbon atoms, forming strong covalent bonds.This arrangement of the diamond is very durable and rigidity, that's why is necessary so many energy to break it.
On the other hand, graphite, has a different geometric arrangement. the carbon atoms are arranged in 2D sheets and each carbon is linked to three other carbon atoms, forming hexagonal rings that are easily broken.