In case there's still any doubt, we don't say, 'to wear one's clothes off' or 'to wear one's clothes down'.
However, you can wear the soles of your shoes down. You could possibly wear a transfer off a T-shirt. Some people might say that they wore the buttons off their shirt.
But we don't use those terms generally for clothes.