With a mistake which was made quite recently, I would be more likely to say "I have made a mistake" but the longer ago the timescale, the more I would be likely to say "I made a mistake + when I made it".
- The plural of mouse is mouses.
- Are you sure?
- Oh, wait a minute. No! I've made a mistake. It's mice.
- I made a mistake yesterday.
- What did you do?
- I told someone that the plural of mouse is mouses!
However, as with all "rules", it doesn't always fit. If I were posting on this forum and I realised that there was an error in my last comment, my next one would say "I'm sorry. I made a mistake in my last comment. I should have said ..."