If you are going to finish your sentence after the word "hours" then only your first one is correct.
The second could be used with some punctuation and some further information:
Yesterday, I had been studying in the library for three hours, when my friend phoned me and suggested we go to the cinema.