We can't afford a car, not to mention the fact that we have no garage.
He has a big house and an expensive car, not to mention a villa in France.
I cannot explain why but these sound better to me. It seems that you use "let alone" if it is followed by a verb and "not to mention" if followed by a noun. That may be coincidence though.