Can I answer "Not mine" or "No mine" when someone asks "Is it yours?"? Is it grammartically right?
These phrases do not contain a subject or a verb, so technically, neither is 'grammatically correct', though 'Not mine' would certainly be a common response in an informal situation such as the one you cite. You are right- 'No mine' is incorrect in response to the given question, though the words could be okay if punctuated correctly and in response to a slightly different question:
1. "Is it yours?" "(It's) Not mine."
2. "Is it his?" "No, (it's) mine."