Both were agricultural laborers, but serfs were "unfree." Not slaves, but they were tied to the land in a way that peasants weren't, exactly.
They had rights, but they had more obligations to their lord and his manor. They had to work a certain number of days for him (for free) and they had to stay in the area (or get permission to go elsewhere).