EXCELLENT question, for which I have no answer except the gut reaction of a native speaker:
"It is high time that we left" and "it is high time that we leave" are equally acceptable, and I am sure that I have said both with equanimity. Taking the present as a basis, therefore, retrogression would give us: "it was high time that we left". Oops! That's not retrogression for the first sentence, it's retrogression for the second. Well, "it was high time that we had left" certainly looks and sounds quite unnatural, so this must be a very peculiar exception indeed.