Yes, it isn't fair. :shock:
"Grammatical" words like to /that/who etc can be labelled differently depending on how you use them in sentences.
For instance "who" can be an interrogative pronoun (as in Who ate the chocolate cake?); or it can be a relative pronoun (as in The boy who ate the chocolate cake is a greedy little boy.)
"To" has two different labels. You have the "to-infinitive" (as in I want to have some chocolate cake.) And then you have the "to" that's a preposition [as in I'm going to the party (where the party is a noun phrase) OR I look forward to hearing from you (where hearing is the gerund and, in fact, what follows hearing is considered to be part of the gerund.
As was suggested higher up, a gerund is treated like a noun. That's why it can be preceded by a preposition. :-D