Elves" refers back to the lines of the poem where the speaker explains "something there is that doesn't love a wall/ and wants it down." The speaker wants to tease his hidebound, prosaic neighbor a little bit by suggesting that some magical critter is making the gaps in the stones. He isn't saying that the neighbor is magical, he is speaking to the neighbor, trying to "put a notion in his head."
The speaker isn't sure what keeps pulling down the stones in their wall, but Frost makes it clear that their meaning is actually metaphorical. The stones, and the wall, are barriers to human connection and conversation, human progress. They are arbitrary and man-made, we keep them up only because it's traditional, or because it makes us feel safer somehow. So the gaps in the wall aren't really made by elves (even though it would be fun to say this to your grumpy neighbor and get a rise out of him), they are made by some innate force that yearns for connection, progress, and interdependence.