This is because viruses bind to specific receptors on the host cell. For example, bacteriophages-viruses that infect bacterial cells, attach to receptors on the surface of bacteria, such as lipopolysaccharides or teichoic acids. Receptors determine the viruses host.
Viruses infect all types of cellular life, but often are specific for the species, so we differ human, animal, plant and bacterial viruses. Viruses that infect plants are harmless to animals and vice versa.