No, that's not a contrary. "See you later" often just means "goodbye". In fact it almost always does in AusE; I believe that's uncommon in global English, but here shopkeepers, checkout people etc. say it without any intention that they will see you later, or even ever again. In other places and situations, "See you later" means "I will see you later", not "I want to see you later".
Therefore, what to say when you don't want to see someone again is a different concept, not a contrary. The simplest way to say it is "I don't want to see you again."
Interested in Language