Hello Panjandrum, welcome to UsingEnglish! (I seem to know the face.)
I don't remember hearing "Second of all"; but if I did hear it, I would take it as a humorous formation. Or perhaps a sign of discomfort with the bare "Second", in someone who had been told that "Secondly" was somehow unmanly and weak (as I once was).
I'll be interested to hear what other members have to say.
I have heard "second of all" used to follow "first of all" enough times so that it doesn't sound strange to me. The only time I've heard "third of all" is when someone is trying to be humorous or sarcastic when making a point. If the speaker bothers to continue to keep count of the points he's making after that, he'll usually just say "third" (without "of all") or "next."