a) There are two possible genotypes because the allele for red flowers is dominant over the allele for yellow flowers, so a heterozygous plant would have the same phenotype than an homozygous plant with RR genotype. On the contrary, to have serrated petals, the only possible genotype is pp.
b) The genotype of plants with yellow flowers and serrated petals must be rrpp, as both alleles are recesive, to express that trait the plant has to be homozygous.
c) The phenotype of a flower with an heterozygous genotype RrPp will be the same of a dominant homozygous flower, in this case, red color and smooth petals.
d) It is possible for the cross of two plants with a RrPp genotype to generate yellow flowers with serrated petals (rrpp genotype), but the probability is low (1/16).