That happens especially when English is not your native language! To improve listening, don't just 'bounce' over to movies and stuff where it is spoken casually, instead why don't you first hear news in English? It is slow and easy to grab, then you can switch over to speeches and finally you will be able to make out the faster-speaking in movies :) Secondly, it's not a problem if you can't make it out - you do know the language, so its nothing to stress yourself with! Take it easy and only 'listen with concentration'!
And one more thing -speak to your teachers and fellows the most you can so you yourself can imrove!