According to the Longman Dictionary of Common Errors,
"The noise of the traffic outside all day annoys me." is not good English and "The noise of the traffic outside all day irritates me." is better.
This dictionary explains that "Something unpleasant that happens repeatedly or continuously over a long period of time tends to irritate or frustrate people, especially because they know that they can do nothing to stop it."
I still do not understand the difference between "annoy" and "irritate." Could anyone explain the difference with some sentence examples?