I don't think it's idiomatic. In English, we tend not to go with logic, but with simple concord, almost based on euphony: yes, it is; no, it isn't.
Using yes to express agreement with a negation doesn't sound natural.
We would say:
No, it can't be, I think you're right.
Interested in Language