Both Dont bother me, or I shall be bothered. and Dont annoy me, or I shall be annoyed. are truisms and sound silly.
As with many English words they can be synonymous, depending on context, but have subtle difference in meaning. Bothered means a disquiet in mental state, if someone seemed reclusive and thoughtful you would ask "whats bothering you?", you wouldn't ask "whats annoying you?" since they show no signs of annoyance.