Self-esteem is a measure of what we believe about ourselves, not what others think of us.
Self-esteem is directly linked to how we evaluate our interactions on a day to day basis, so the stress that we feel is very often tied to whether we believe we're "enough". Stress is a major factor in a variety of physical ailments; it directly affects blood pressure and the immune system.
Self-esteem also is a direct influence on a person's interest in taking care of themself. If I believe that I'm cr@p then it naturally follows that there isn't much point in eating right, exercising, attending to my spiritual health, having friends, ... People with low self-esteem are much more likely to be chronically depressed