1 I started out by watching TV. Then I did some gardening. Then I read for a while. After that, I have to go back to watching TV. ["go back to + verb-ing" means 'revert to a former practice'.
2 I used to have a portable TV, so I didn't have to go back home whenever I wanted to watch something. But they've changed the frequency of the broadcasts, so that reception on the portable one is very bad - I have to go back to watch TV.
So - and this is the answer I imagine you're looking for - when 'go back to' relates to the other verb, use the -ing form. When 'go back' refers to actual locomotion use the 'to' infinitive.
Student or Learner