We normally 'answer that', not 'answer a person that'. If it is an answer, it is clear who it is addressed to.
'That' is usually not omitted with 'answer'.
Incidentally, there are twice as many sentences with 'replied that' in COCA than with 'answered that'.
- For Teachers