'In your face' is not the nicest thing someone could say to you! In fact, it's another way of mocking by suggesting that you have been wronged and there is nothing you can do about it. It's a lexical (word) way of pointing a finger at someone and laughing with arrogant confidence. You might want to try ignoring the taunt.
Student or Learner