Note also that 'didn't have to' can be used sarcastically with a different meaning - you did something. and I didn't want you to: 'You didn't have to tell her.... Now we'll have to start all over again with someone else. And this time, keep your mouth shut.'
This sort of sarcasm affects other modals:
Teacher: You should have been here half an hour ago.
Late pupil: Why Miss? What happened?
Yes, it is all part of the example - giving the context (of displeasure/annoyance) behind the usage. (I was a bit lazy about explaining the context - in fact, I left it to your imagination!)