'Baby' is used informally and can be offensive. It is often used to decribe an attractive woman. On the other hand it can be used towards a person of whom you are fond.
So, used while speaking to someone you are familiar with is acceptable but used to strangers or subordinates it is offensive. It is normally used this way by a man to a woman. It can be used in an abstract way as well. If you were watching a horse race for example, you might be inclined to say 'Come on baby' in urging your horse to win.
"Come on baby, I didn't mean to come home drunk' might be used to get out of a tricky situation.
Come on baby, do you think I'm stupid, of course you meant to come home drunk. Might be the reply