Yes, people use the word differently.
However, I have honestly never heard someone say to a man who is out without his wife and children "Oh, is your wife home babysitting?" She's just with the kids. However, when a woman is out without her husband and children, they say "Oh, George is babysitting?"
And likewise, when "George" is out, it's very rare for the mother to say "I have to babysit while George is at his bowling league." She's just home (or out) with the kids. But when "Mary" is at choir practice, it's much more likely that George will say he has to "babysit." (Your story is the ONLY time I've ever heard a wife say she will babysit while her husband does something.)
So I concede that people will use the term to mean they are watching their own children, even when there are not sexist overtones about whose "job" it really is and who is just "helping out."
However, I maintain that a child old enough to be taught to drive does not have a babysitter!