In most cases, it's precisely as you think - it depends on the position of the speaker. But have a look at the dialogue below:
[woman talking to her man on the phone]
Woman: Mike, I'm missing you so much. I just can't wait for you to come back.
Man: No worries, honey. I'll be coming (not going) back soon.
Woman: You promise?
Man: Sure, babe.
Mike is away from his home, and yet uses the verb come instead of go. Interesting, innit!