Would it be correct if I use the bare infinitive ' let's go swim' instead of 'let's go swimming'?
I'd say either
Let's go swimming
or
Let's go for a swim.
I'm sure you could hear let's go swim. But that doesn't necessarily mean it's correct to say so.
Hope this helps
You have asked similar questions in a couple of other threads. I expect the answers here will also be similar.
I would say only, "Let's go swimming".
Context is always important; labelling is rarely important.