The answer is: Many voters dislike having to declare party preference.
Whether we like it or not, most people tend to judge other people based on party preference.
This often lead to conflict that happens on various social settings which make many people prefer to keep it for themselves.