I think it helps to look at an example.
Let's say we had the set {1,2,3,4,55}
The median is the middle most number. In this case it would be 3. There are the same number of values on either side of 3.
The mean is found by adding up the numbers and then dividing by 5 (which is the number of values). The mean is (1+2+3+4+55)/5 = 13
Notice that the median (3) is much closer to the main cluster {1,2,3,4}; whereas the mean (13) is trying its best to be somewhere between that main cluster and the outlier of 55. The mean is pulled by the outlier as if it was a gravitational or magnetic pull.
As such, the mean is a bad measure of center when outliers are present. The larger the outlier, the worse the representation. This is why home prices for instance rely on the median. Multi-million dollar mansions greatly skew the mean housing price to be (much) larger than it should be.
If there aren't any outliers, then the mean is a better measure of center because it involves each item weighing in.