Khim, welcome. It doesn't necessarily follow that being good at math means that your poor at language. However, I'm not very good at math. Mind you, you could try to apply that ability to your English.
It's easy to make comments about cofidence, much much harder for the person who lacks it to deal with. however, you must remember that if you get your message across, then you have comunicated successfully. That is more important than worrying about perfection. Also, when you are speakin g a foreign language, you are doing the hard work, so there's no reason for the other person to feel clever if you make a mistake. Try not to worry about mistakes and focus on getting your point across.