False.
The first time, when the small towm 20%, the decrease is calculated over the intial population. So the lost is 20% of initial poulation = 0.2 * initial population. And the new population will be 0.8*initial population.
Second time, the increase is calculates over the new population. The gain is 20% of then new popultion = 0.2*new population = 0.2*0.8*initial population = 0.16* initial population.
Then while the loss was 0.2 * initial population the gain was 0.16*initial population, being the gain less than the loss. So, at the end the town's population was less than before the downturn.
When you use percentages you have to be aware that the base over which they are calculated determines ther result. A lower base turns into a greater %.