No. It does not mean that he is the best swimmer out of all of us. That would imply that he can swim better than all of us.
It means that he can can swim just as well as any of us -- and that includes our best swimmer, regardless of level. He can swim as well as the best swimmer.
Now, that is the literal meaning of the sentence. In reality, when you say he is as good as any of us, it could imply that he is as good, and possibly better than all of us. It may be what you're thinking in the back of your mind. That's how the sentence can come across, depending on context.
So, your intuition serves you well.
Student or Learner