That sounds great.
a) It is high time we
went/left.
=> We've spent too much time here already, so it's high time (that) we left.
=> We're fed up with what's going on here, so it's high time (that) we left.
=> We're scared about what's going on here, so it's high time (that) we left.
high noon, for example, is when the sun is at its highest in the sky;
high time, which is similar in meaning to
about time, is when time is at its worst; i.e., an event is way overdue, be that event an exit as in leaving a location or something like, say, a chore:
EX: Sam, it's high time you did your homework. You've been putting it off for a week.
b) It is high time for us
to go.
=> This means the same as a), with the exception that "us" being an object pronoun, expresses a group is undergoing the event; that is, they are not the actors or initiators of the event; someone or something is making "us" go; it's not our decision, and yet, it could be our decision, but we're using "us" to sound more polite, and less direct. "We" is direct. It expresses exactly who the actor or the decision maker is. If you use "us", it makes it sound as if you have no part in the decision, which even if you do, makes it a polite way of leave taking.
So technically, "we" and "us", not "went" and "to go", should be highlighted in a) and b) respectively. As for the infinitive verb
to go, a preposition heads the phrase, and the reason it lacks tense.