I'm not a teacher, but I think it's a cultural difference. I know some Chinese, so if you write the expression said by Chinese with some explanation, maybe I can explain it to native speakers.
In Korea, we call such spouses sometimes "You! womanizer" or any swear word you want to say depending on your mood. And I guess native speakers will say various things depending on their mood or how serious the affair is.
As Chinese and Korean share the same Chinese characters, I might be able to understand your expression.
Student or Learner