We do love our double negatives in this language. I think, if indeed your were left standing all alone and nobody did come, you would shake your head sadly and say, "No."
Alternatively, you could say (just as sadly), "You're right."
But to simply say, "Yes" would confuse him.
Interested in Language