yes you have the right feeling about the phrase. However, you can use it with unfamiliar people, as long as the conversation is friendly and familiar. You usually use it when somebody has 'jibed' you with a derogatory phrase meant as a joke, so its unlikely to come up unless you are in this situation.
In a formal context, you can "return the sentiment," or "feel the same way"
Can the expression "Back at ya" only be used among good friends without sounding vulgar or can it be used anytime anywhere?
I just want to comment on your use of the word "vulgar". If you were using that in the sense of "informal" or "of the masses", it is correct. However, the word is mostly used to refer to things that are indecent.