Let’s pick a paperback and a hardback
There are 4 x 6 possible choices
Now pick any three of the remaining 8 books
8C3 = 56
This means there are 4 x 6 x 56 = 1344 selections of 5 books that will contain at least one paperback and one hardback
8C3 means picking 3 from 8 without worrying about order
8C3 = 8 x 7 x 6 / (3 x 2 x 1) = 56