The only use I know of "whisk off" is when someone takes someone else away on a (sometimes surprise) holiday.
Sue: What are you doing for Jane's birthday?
John: I'm whisking her off to Venice for a long weekend.
Sue: Wow, lucky girl!
It has absolutely nothing to do with brushing or wiping. Where did you get the idea that that was the meaning?