In a free enterprise system, embodied within a market economy, the type of literature that predomines depends not on the system itself but on the preferences and income of the consumers and on the abilities, skills and willingness of the suppliers.
Therefore, if most of the people is demanding trashy sensationialist novels because they prefer them instead of literary masterpieces, the first ones are what the suppliers (writers) should produce if they want to sell books. They can produce good quality works, but nobody would purchase them if they do not correspond with their preferences.
It can also be the case that consumers like masterpieces but these are too expensive because they require more effort than trashy books, and they cannot afford to buy them.
Therefore in a market economy, in order to produce high quality literature works, these need to be appreciated by consumers and affordable too. The system itself has not the power to prevent the flourishing of amazing works of literature.