'Went' isn't the past but present subjunctive from 'go'. The past subjunctive is 'had gone.'
Neither should, nor must, nor ought can be preceded by the particle 'to'.
'Must' with a perfect infinitive does not express advice or duty. It indicates supposition. 'It must have happened = I'm sure it happened.'