The color of the rocks on its own can not be considered as a relevant factor in determining the type of the rock.
The reason for that is because the same color can be find in multiple different rocks, so it can be really tricky to rely only on the color and not take anything else in consideration.
Another reason, more important one, is that the outside part of the rock and its coloration can be easily affected by multiple things, be it exposure to certain type of soil, decomposition of plants, oxidation, rain, so the color we can see on the outside can be misleading and different from the actual color of the rock. In that case, it will be much more precise to break the rock, or at least little part of it, to see the color of the rock on the inside.