The first speaker is possibly thinking something like, "If I had been able to afford to go..., I would have gone, but I am hard-pressed for money ( a situation that is true around the general present, which extends back to the time of deciding not to go)'.
The second is possibly thinking, " If I had decided... I would have missed the (later) opportunity to be (now) with you".
In speech we often run thoughts together.
Student or Learner