But this could only be a reply to someone who claimed that it was rather late. So, you can't replace 'rather'.
Man: "I know it's rather late to be apologizing ..."
Woman: "It's not rather late. It's too late."
- For Teachers