A 'pit' is like a hole in the ground.
The expression 'bottomless' means never-ending. Something that has no 'bottom'.
So, to say 'The bay was like a bottomless pit' is to say it had no ending, that it went on forever, that it couldn't be measured (not literally, just as a figure of speech). When you look down a dark hole sometimes you wonder where the 'bottom' is, where it finishes. Writers use this idea to convey a sense of mystery and depth to a scene.
Student or Learner