The total amount of books the bookcase can fit is given by the product between the amount of shelves(5) and the amount of books each shelf can hold(10).

Now, we know that the whole bookcase can hold 50 books.
The books that won't fit the bookcase are the difference between the amount of books Samson has and the amount of books the bookcase can hold.

3 books won't fit the bookcase.