If I get it right what you're trying to say, I would modify your sentence as follows
'Every morning I would look forward to going back to work because of all of the fun I used to have there'.
Disclaimer: Not a native speaker, just have been living in the States for 2 years. This answer was mostly being driven by my subconscious, not the formal grammar rules