The expression "It is (high/about) time + past verb test" is used to complain about or criticise something or someone:
It is time that the government took action.
It is about time that the government took action.
It is high time that the government took action.
The expression Its time we left would express an implicit criticism of our action because we must have overstayed our welcome. If you just want to express the idea that we need to leave now because we have other obligations and cannot stay, then you would simply say: It's time for us to leave or it's time to go home .