I am not an English teacher, just an ESL student who has faced with similar issues over his learning process. I will give you some advice I have received from my teachers.
They say the main issue about not understanding what someone else is saying is the lack of vocabulary. Of course, there are other areas to take into account such as grammar, yet when it comes to bringing your listening skills to the next level, improving your vocabulary is the area where your focus should be on. Curiously, the most effective way to improve vocabulary is by reading. Therefore, to improve listening my teachers recommend me read lots of stuff related to the subject I want to listen afterwards. If that were sports, read sports and so on. Of course, you need to complement your vocabulary learning how these words sound, and for that, you might use the google translate listening function or other more sophisticated programs. Additionally, when reading you are also more aware of understanding grammar, which is commonly passed over while listening given that ESL students listen for content and not for details. Try reading.
Another really important consideration is to have the audio transcripts along with you to check those words you did not get. And again, by reviewing them you are also improving your vocab. There are plenty of websites on the Internet where you will find listening exercises and their transcripts.
Finally, with respect to speaking - just speak. It’s a matter of time, the more you practice, the more confident you will be and hopefully getting your message across will be a lot easier.
Student or Learner