Is it correct to say:
John talked with Mary on the phone
Mary: What are you doing now?
John: I am going to swim with Peter now.
John is not answering Mary's question exactly, but in telling her what he is on the point of doing, he is delivering an effective response.
If we get in to the area of implicatures, your telephone dialogue can be interpreted as:
Mary: Are you free now?
Another personal preference; I would probably answer the question in one of these ways:
I am just going to go swimming ..
I am about to go swimming ..
I am just off swimming ..
When speaking of the leisure activity, it is more usual to go swimming thanto swim. Note that I wrote more usual, not more correct.