In AmE, "ma'am" is always appropriate when addressing a female over the age of majority, although it's preferable (from the female's point of view) to be addressed as "Miss" until she's about 30 years old. (It is often a bit of culture shock for American women - the first time they were addressed as "Ma'am" instead of "Miss" by someone. Made them feel old. ;-) ) But in the US "Sir" and "Ma'am" are common forms of address for adult males and females. It can be used as both a sign of respect (as when speaking to a female judge in court and saying "Yes, Ma'am") or when confronting a stranger ("Ma'am? Are you waiting for the Woodward Express bus? Do you know what time it will arrive?")
So, depending upon your location and the particular culture, I can't imagine "ma'am" being inappropriate when addressing a supervisor.