The problem is that there are many, many dialects and different accents (not just "British" and "American"). Dictionaries usually give [æ] as the pronunciation in words like "man" because that is close to the pronunciation in both Oxford English and General American.
In real life, though, few speakers will use exactly that sound. Most British dialects will use a sound much closer to /a/ or even /ɑ/, while some American dialects may use something more like /ɛ/ or /e/. A few American speakers might even pronounce it as a triphthong, something like /ɛɪə/, although this is probably rare. This is part of what is called the "southern drawl", because it's typical of speech patterns found in southern areas like Louisiana or Texas.
The examiner might have thought you were saying something like "slong" (/slɑŋ/ in some dialects), and that rhymes with a, er, *ahem* rude word. (But don't worry: you found a way to explain what you wanted to say, and that's an important language skill.)