"Oh, bother!" was a phrase used in older times; it was something a character in Pride and Prejudice or Jane Eyre might have said.
"Oh, brother!" is the common expression used today. It can mean "oh, dear" or "oh, boy" or "woe is me" or "Oy vey!"