I would choose "fond" because it makes your meaning clear. If you say that you don't have good memories, the reader might conclude that your memory is unclear rather than unpleasant.
"I don't have fond memories of her because of a bad experience"
"I don't have pleasant memories of her because of a bad experience"
are better choices to clearly convey your meaning.