Not a teacher.
I think you should use past tense in this situation because:
- the interviewer use the past tense to ask you
- you don't say when did you get there, but the interviewer can still understand that the time is near from now (you couldn't come there from yesterday), you had to know the appointment time before to get there on time.
Beside, based on my experience, they often use past tense although the time isn't mentioned in AmE.