They plot out a given area, such as 500 acres. They pick a section, maybe 100' X 100' square, and set live traps.
After catching the rabbits, they do a count and note sex, age, health. and size, then turn them loose. The number of rabbits caught, times how many 100' X 100' sections there are in 500 acres, gives them the approximate amount of rabbits in that 500 acre area.
This kind of thing is done for most animal populations, except for those that can't or shouldn't be caught. Those animals are often counted by observation or by photography and then multiplied by the given area as I said.
-- From Google.