The short answer is when to is NOT a marker of an infinitive but a preposition. In I want to go home, to go is an infinitive and not a preposition and therefore the base form of the verb go is used after it. In I look forward to seeing you, to is a preposition and hence the ing form (gerund) of the verb see is used. A gerund is verbal noun or verb-noun. It is governed by the preposition to and takes you as the object.
I like reading (* to reading) poetry.
I like to read (* to reading) poetry.
I have an aversion to swimming (*swim).
Student or Learner