Probably, molecules responsible for holding cells together (cell-cell adhesion) are specific for species, meaning that each species has its own molecules that don’t act on other species’ cells.
The term used for this property is species-specific cell recognition and most likely involved molecule is proteoglycan-like complex also called aggregation factor.
Even though, sponges are simple and at low phylogenetic position, their histocompatibility systems (compatibility of tissues) are at high level.