The thermal conductivity differences between milk and water might be a small factor; however, given the large differences, I'm guessing the following are more important:
1) Starting temperature differences
2) Differences in the weight of the samples
3) Differences in container shapes, volumes, surface areas
4) The freezing point for your water samples (unless it was distilled water)
5) The heat of fusion difference between the milk and water