yes you have the right feeling about the phrase. However, you can use it with unfamiliar people, as long as the conversation is friendly and familiar. You usually use it when somebody has 'jibed' you with a derogatory phrase meant as a joke, so its unlikely to come up unless you are in this situation.
In a formal context, you can "return the sentiment," or "feel the same way"
- For Teachers