I agree completely.
Environmental factors control evolution through surivial of the fittest, or natural selection. This is not nearly as much the case with humans as wild animals, or early on humans, but it does still come into play. For example, lactose intolerance was erradicated in areas where milk was a primary source of nutrients during a famine. The famine was environmental, causing natural selection to elimate lactose intolerant people in the area.
A more basic example is fur color of certain animals. They were selected by their environment to survive and pass on their genetics, causing the species to evolve.