"No" = "No, I do not mind; I have no objection"
"Yes" = "Yes, I mind; I have an objection"
In practice, must of us would use tone, gesture, or additional words to make our meaning clear:
"No, go ahead."
"(Yes.) Sorry, it's taken."
"Yes", especially without explanation, sounds very abrupt.
- For Teachers