I'm a native speaker and I get confused by the phrase 'not to say'.
One or two of his essays are purely informational (not to say weird). One of my favorites is essay 49, which reads in its entirety: "You should never put the new antlers of a deer to your nose and smell them. They have little insects that crawl into the nose and devour the brain."
Is 'not to say' a way for authors to imply something by explicitly 'not saying' it?
Yes, I think that's it exactly.
I would say "weird" if I weren't too polite to say so. So I'll say "not to say weird" instead, so I can say it, but at the same time say I didn't say it.
Some of his more colorful (not to say out and out racist and misogynistic) views are expressed in most recent speech.
I'm not a teacher, but I write for a living. Please don't ask me about 2nd conditionals, but I'm a safe bet for what reads well in (American) English.