I prefer the definition: any verb that is composed of two or three words that act as a unit to create an idiomatic meaning. Some people see any verb followed by a preposition as a phrasal verb. Others say that phrasal verbs can only be verb + adverb, because verb + peposition is a prepositional verb. By my definition "go to" is not a phrasal verb.
When it comes to infinitives, you have to use the base form of a verb.
If you want past tense, you have to indicate it another way. The infinitive form of "went to" is "to go to".