Would rather expresses a wish that is unfulfilled in the present. It can be also used to express wishes for the future that may or may not be fulfilled. SO, you use this structure with the past (told and left as you said), but the meaning is present or future.
I read it's less common, but if you want to express a wish unfulfilled in the past, you use past perfect:
I would rather she had gone to London last week.
Would rather (do something) can also express a preference, and then it goes with the infinitive without to:
I'd rather go by car. (= I'd prefer to go by car.)
Would you rather have tea or coffee?
I'm tired, I'd rather not go out this evening.
I'd rather cook the dinner now.
I'd rather you cooked the dinner now.
Student or Learner