It's not stupid at all!
While the exclamation "Jesus!" is used by some, others find it offensive (sacrilegious). "Jesus Christ!" is even worse. So many others things have developed with the same initials. I say "Jeez!" quite often and "Jeezum Crow!" when I'm really perplexed. So, Jiminy Cricket is just another way to take what would possibly be offensive (and inappropriate for a child to say) and make it okay.
I've never actually heard anyone say that OR "Christopher Columbus!" but people will be creative at times. My favorite ever was "Oh, for the love of Santa and all his reindeer!" Much less likely offend than using the lord's name inappropriately.
Student or Learner