I can sympathise completely. I speak (well, I used to speak) almost fluent French, having been brought up by two parents who were pretty much bilingual. Nonetheless, I have never been comfortable with French over the phone. I put it down to the fact that I have realised that I lipread when people are speaking to me (I'm not deaf, by the way!). I have no idea why that is but, to this day, I will do anything to avoid having a telephone conversation in a foreign language.
I realise that doesn't help you solve your problem but I thought I'd at least show that I share your pain.
Student or Learner