The fact is that he is having a party next Sunday.
writing
- (Tomorrow,) He is going to invite me to his party next Sunday (tomorrow)
- (Tomorrow,) He is going to invite me to his next Sunday party (tomorrow)
I'd use the first, and I think it's clearer and easier to understand if you put tomorrow at the beginning.