i think we use answer a question or answer me whenever answer is used as an infinitive object.Like:-
He refused to answer me.
He refused to answer the questions.
In other situations we use 'answer to'.Like:-
It is not possible to know the answers to all the questions.
So see the difference:
1)It is not possible to know the answers to all the questions.
It is not possible to answer all the questions.
2)He gave answer to my question.
He cannot answer my question.