Speed is the distance you travel per period of time. (s = d/t). So since the time is in the denominator of the fraction, (lower denominator equals higher number), a lower time would mean a higher speed.
Think of it this way. The person who ran 1 mile in 5 minutes is faster than the person who ran 1 mile in 10 minutes, because they have a shorter time.