1. Most likely, someone did something nice for/to you, so you want to do something nice for/to them.
2. You can look up the word curtsy or curtsey, a kind of greeting by a girl or a woman that could be returned. They're variations on a word derived long ago from courtesy.
I think most Brits would be rather puzzled by that phrase. we are used to saying... Return the compliment (if someone said something)/ return the favour (if someone DID something/ . Courtesy to us means Politeness. You don't think of having one of them- A Courtesy.
True, but you don't think of having a compliment either, even though, as you say, you can return one - which raises the question: If someone gives you a compliment, and you give the same compliment back, do you still have it?
Return the courtesy sounds OK to me. Say there was some reciprocal arrangement. Like you come to my home town/country and I show you around and play host, etc. When I visit your town, you return the courtesy.