Yes, to find the distance between two points in one dimensional plane ,it matters that which way you subtract the values.
For, example we have to find distance between , a=6 and ,b=23 on the number line.
⇒a-b=6-23=-17, gives distance in Negative which is not possible.
⇒Now,b-a=23-6=17, gives distance in Positive Units .
⇒So, it is advisable that you subtract Smaller number from Larger Number. Or Take Modulus of Difference between two numbers which always yield a positive Quantity.
Now, Coming to Two Dimensional and three dimensional Plane
Distance between two points is given by

So,In Two Dimensional and three dimensional Plane , Direction does not matter that is whether you are taking

because you have to square the difference , and then take it's square root which always yields positive value.