Let's first count how many atoms of each element we have on each side of the reaction.
Reagent side
Mg --->1 atom
Fe ---> 2 atoms
O ---->3 atoms
Product side
Mg --->1 atom
Fe ---->1 atom
O ----> 1 atom
We see that we have three oxygens on the reactant side and one on the product side, therefore we place the coefficient 3 in front of the MgO molecule:
Now we balance the Mg atoms that changed, we have 3 atoms in the products, so we put the coefficient 3 in front of Mg
And finally we need to balance the iron, we have 2 iron atoms in the reactants, so we put the coefficient 2 in front of the iron molecule:
Now we have on each side of the reaction:
Mg ---3 atoms
Fe ----2 atoms
O ---3 atoms
Now the reaction is balanced.
We have a substitution reaction because Mg trades place with Fe