The phrase "o'clock" is passed down from its origin in British English. It's usually used after whole numbers indicating time.
I think it's an English idiom moreso than anything having to do with English grammar. You can thank the British for developing the habit many centuries ago and passing it down to us. lol