Yes, the caption is a pun on the idiom. He picked her up (swept her up in his arms) and carried her over the puddle.
The original idiom suggests that when you are suddenly "swept up" in the early emotion of love, you may feel light-headed, giddy, faint. Your heart is in your throat would be another way of saying it, perhaps. Dizzy, certainly. Put all those physical reactions to this strong emotional attraction and you are "swept off your feet."
There is also a background image of the knight in shining armour atop his charging white destier, reaching down to sweep the lovestruck damsel out of the haystack and into his arms as they gallop off into the sunset.