I am not a teacher.
I am not aware of any rule other than ease of pronunciation.
In all of the examples you give of words without the 's' try to pronounce them with an 's' and see whether that makes it easier or not to say them. Then pronounce the 's' words without the 's' and you will see that none of these words flow as they should.
I imagine 'portsman' is supposed to be 'sportsman'.
Student or Learner