Hello Caz, welcome to UE!
1. ??We go to swim every week.
2. We go swimming every week.
3. We go to swim in the loch every week.
4. We go swimming in the loch every week.
#2 would be the usual form; but where the location is specified, you might use an infinitive, as in #3.
However, even then, #4 would probably be more common.