I have a special affinity for you.
Is correct. 'Affinity' here means a spontaneous, natural liking for someone, (or something, such as a particular kind of music).
'affinity to' is used for the second meaning of the word: a similarity of characteristics suggesting that two things are 'related' somehow. This applies especially to animals, plants, languages, buildings. We might say that a new building has no affinity to contemporary architecture, meaning its design is not in line with/ a further example of modern contemporary architecture.