Answer is: yes, polarity affects the rate of diffusion.
For example, in phospholipid layer in cell membrane, there are hydrophilic heads (polar part of the layer) and hydrophobic tails (nonpolar part of the layer).
1) Small, non-polar molecules diffuse easily through this layer, no energy is required, this is passive transport.
2) Polar molecules cannot diffuse through phospholipid layer using passive transport, because hydrophobic tails does not allow that (polar molecules are rebounded), energy is required for that, this is active transport.
Also, the lighter (smaller) particles move more quickly than heavier (larger) ones at the same temperature.