That's a good question. There is an alternate way of saying that, which I think is now more old fashion. "...of our having to..." in which case it would clearly be a gerund. I would tend to say it is still a gerund. I have read about this in House and Harman, but I can't remember at the moment. I think the discussion was about why "us" is in the objective case when it is the subject of the gerund.
I'll look into it.