In my opinion, you have to start at some level. If you think you are having a hard time understanding native English, say, 90% of the time, then start listening to a material with a slow speed. Repeat what you are doing every day. If you feel comfortable to move up a level, then try to listen to a faster audio, and keep on listening to that until you feel comfortable, then move up a level. It is a repetitive process, but trust me, it works. I used to find it very difficult to understand fast English, and it was so frustrating. But I had a goal, and I didn't want to give up. So even if it was a painstaking process, I kept on listening to native English until I reached a comfortable level.