On a small island, it is unlikely that outside sources of food could be reached (or at least reached quickly.) If the soil is poor, crops will fail, which cuts off a prominent food source for every animal on the island. Therefore, not only will your crops fail, any animals that might have been on the island for the use of food or work will die.
Hope this helps!! <3