Not a teacher.
There are some with far more experience in this area than I have, and I'm sure they will eventually gravitate over this way. In my dealings with the British there have been mostly difference in pronunciation, spelling, and phrasing.
The pronunciation is the most obvious part, and it makes it seem incredible that they and I are actually speaking the same language sometimes (especially Scottish accents!). We pronounce our R's a lot more than the British do, which is what most Americans notice first. Most Americans consider the British accent to sound more sophisticated and "superior" to our own. Sometimes I get the image of a snobby aristocrat looking down his nose through his monocle, as he strokes his thick mustache when I hear some British accents.
The spelling isn't that different, mostly just some E's and U's that the Americans have taken out for whatever reason and some letter switching. Like you pointed out: colour instead of color, armour instead of armor, theatre instead of theater, and the use of other useless letters.
The British do have a great number of different phrases than the Americans do. Lots of words mean different things in either accent. Some British idiomatic expressions are as foreign to me as most English idiomatic expressions are to you! I learned just the other day that "jammy" means "lucky." Some guy called me a "jammy bugger" and I had no clue what he was saying. I've never had English "pudding" either.
I wonder why it is that Americans and Canadians (pretty similar) formed fairly different accents when compared with Australia, New Zealand, and South Africa? I would even say that Australia and New Zealand are more remote than the US, but the accents are still close to British (at least to an American). Is it the time periods the different areas were colonized? (Question directed at anyone who knows.)