Knights were indeed lower than vassals and held less prestige in medieval society. It's not true though, that knights didn't have enough prestige in medieval society. Vassals swore to the king or the ruler of a region, whereas knights often swore to vassals, so they still had people under them, but they were not at the top.