It was about slavery. Whigs were divided on the slavery issue internally and started going their own way due to support of different causes. Some like those in south didn't believe that slavery was a problem and wanted to focus on prohibiting immigration so they went to the know nothing party. Others fought slavery so they joined abolitionists.