Any two objects when touching each other will try to reach the same temperature. Around your body, this means your body heat rushes out to warm the cold air around it. You feel cold because you've lost heat, not because the cold air has transferred any "cold" to your body.