I think "having him back" implies something on-going, which does not suit the context. It is a one-off happening.
The gerund works in other contexts but not this.
Yeah, I know. What I meant was to replace "to have" as a whole by "having", not simply "have". tedmc has already answered my question, but thank you anyway.