I'm assuming that you have to concatenate the toothpicks one after the other to build the sides of the rectangle.
Recalling that the opposite sides of a rectangle have the same length, the formula for its perimeter is

Since we have 12 toothpicks, we have

So, the problem reduces to finding all the possible couples
such that

It's easy to list all the possibilities:



The reimaining possibilities are just the reflected versions of those we already listed, so the 1-by-5 rectagle yields the existance of the 5-by-1 rectangle, and so on. It may even be debatable is these two cases should be considered as the same or not.
So, there are five possible rectangles in total (one of which is a square):
1-by-5
2-by-4
3-by-3
4-by-2
5-by-1