In practice, context and, in speech, tone of voice and emphasis, convey the message. Also, we rarely give a single word response. Father's actual response are more likely to be:
No, we're not. / No, we aren't.
Yes, that's right.
My personal feeling is that the 'no' response is more natural and common.
Student or Learner