The simple answer is - Forget 'not'. If 'not' seems illogical to you, then it probably is to most native speakers. Such utterances are more natural without 'not'.
A longer answer would involve a discussion on (pseudo-)diffidence in dated upper-middle-class English, which would be irrelevant to most learners today.
- For Teachers