Make sure you are looking at a infinitive and not a prepositional phrase by viewing the word or words that appear after the word 'to' in the sentence.
If the infinitive begins a sentence and is followed by a verb, it functions as a noun. Whether following a verb or noun, if the infinitive ends a sentence and could be moved to the beginning of that sentence by adding a comma, it functions as an adverb. If the infinitive ends a sentence but can not be moved to the beginning of that sentence