Not only Muslims speak of "people of the book"- this phrase is also known in Judaism (but in Judaism it only refers to the Jews).
From the point of view of Muslims, this phrase refers all the Abrahamic religions: it's Jews and Christians - the worshipers of God who don't acknowledge Muhammad.