If someone repeatedly calls me to know when I will be there, and I get irritated with his behavior, can I say "Stop being fidgety. I will be with you soon."
A little too "in your face".
Try "I am sorry to keep you waiting. Please be patient a little longer. I'll be with you as soon as I can."