There's nothing wrong with doing what I did- just because a question is passive, it doesn't follow that the answer has to be. Wouldn't the passive answer be 'not by me', which I could have used, but I don't see why there has to be the link you're suggesting between the question and the answer. The answer is about passing over information in response, not following the grammatical forms used in the question.
