It is true that you must not say "go to __________", when you are giving someone directions by describing an absolute location. Because by saying such gives a vague instruction. To make it more absolute you can use directional words such as ‘to the left’, ‘at the right side’, etc. and more landmarks.