It depends on the tense.
Have to implies present tense.
Had to implies past tense.
If you use 'have' you'll be changing the meaning of the sentence.
'Must' would've also been an appropriate choice, because in that structure, it also refers to present tense, the same as 'had to'.
To me, that sentence sounds strange though anyway.
The sentence is obviously directed at a second person, given the use of I'll.
Honestly, I would've had it.
"I had to hurry or I'd be late" for past tense.
"I have to hurry or I'll be late" for present tense.
Given the choice though, 'had to' is the most appropriate.