Since preterite and imperfect are both past tense, it can be confusing figuring which to use in different scenarios. To know when to use me encantó versus me encantaba, it helps to know the tense (as you do) and what the different tenses are used to express.
Me encantó is preterite; preterite is generally used for past actions that are completed. In the example you provided ("I loved to visit"), your visiting another place was an action in the past that is seen as completed as of now.
Me encantaba is imperfect. This tense is used to describe past continual actions that were interrupted by something. In the case of me encantaba, if you were to say Me encantaba ir a la escuela, you are describing something in the past that you used to do continually (go to school), but don't do anymore.
Consider the following sentences.
Me encantó visitar.
Me encantaba visitar.
Both of these sentences make sense, but they have different meaning. The first sentence (preterite) implies you visited somewhere, you enjoyed it, and now it's done. The second sentence (imperfect) implies you would continually visit somewhere (weekly, monthly, annually, whatever), and you enjoyed it, but for some reason you don't continually visit the place anymore (it was interrupted by something).
I hope this helps you; good luck.