"Not really" is a tentative or polite or undemonstrative "no" to a preceding question, e.g.
1. "Do you like oysters?" "Not really."
— i.e. "No, I don't like oysters; but I don't want to say so too forcefully."
Or it may express the truth of a situation, as the speaker sees it, e.g.
2. "Did Zéphirine seem interested in the job?" "Not really."
— i.e. "Zéphirine attempted to give the impression that she was interested in the job; but I could see that she wasn't really interested."
It may also express incredulity, as a question, e.g.
3. "Zéphirine decided to take that job as a toilet cleaner." "Not really?"