It's no different from "Nice clothes are wasted on me" or "Sarcasm is wasted on some people".
It's not a case of them wasting what they have been given. It's the fact that it is a waste of the time of the person doing it, to bother doing it to those people.
"Sarcasm is wasted on my brother". He doesn't waste sarcasm. It's a waste of my time being sarcastic towards my brother because he doesn't understand/appreciate it.