When you say "didn't have to", does it mean "You didn't have the need to do something, but actually did it" or "You actually didn't do it"? Doesn't it have to be corrected to "You shouldn't have done that"?
A: I baked some cookies. Why don't you have some?
B : You didn't have to bake them. I already bought some.
Our instructor told us we didn't have to do that exercise, but I did it anyway.
Our instructor told us we didn't have to do that exercise, so I didn't.
'Shouldn't have done it' is another kettle of fish, I think. You shouldn't have done it implies that you did it, and if you didn't have to do it is used synonymously there, then it does indeed also mean that you did it.