Here is the first meaning that occurs to me:
1. He studied so hard that he could pass the exam.
— "he studied so hard that it is possible that he will pass the exam".
The first meaning you suggest, i.e.
2. He studied so hard that he was able to pass the exam.
may have been possible in earlier forms of English, when "could" had a stronger "past tense" meaning; but I don't think it would occur to a native speaker now. (Though I'd be happy to stand corrected, if someone else does find it possible!)
There is another (obsolete) meaning to the sentence:
3. He studied so hard (so) that he could pass the exam.
In earlier forms of English, it was possible to use "that" in such a context with the meaning "so that". Again, this is no longer a natural usage.
Does that help?