***** NOT A TEACHER *****
Good morning, Sahil.
(1) Don't feel bad. We ordinary native speakers don't know all the rules,
(2) Of course, the most important thing is to speak (or write) correctly.
(3) Many students learn all the rules, but they still can't speak correctly.
(4) Yes, I believe that you were correct in saying: I am sorry TO HAVE
KEPT you waiting.
(5) As you undoubtedly learned in school, this is an infinitive (because of
the "to" and the perfect because of "have kept." ) My grammar books
call this a "perfect infinitive."
(6) Suppose you are the boss. Someone comes to see you. Your
secretary tells that person to wait. That person waits for 45 minutes
because you are very busy. Finally, you walk out of your office and say
to that waiting person: Oh, I am so sorry to have kept you waiting."
(a) I think that most books would suggest the perfect infinitive because
it refers to the past (the 45 minutes that the person was waiting) and the
present (when you walked out and started to speak).
(b) If you had said, "I am sorry that I kept you waiting," some books
would recommend that you use the past only for something that is
definitely past and finished. For example, if you call him/her the next
day, you would say: I am sorry that I kept you waiting so long yesterday."
(6) It's great that you speak correct English even though you cannot
cite the rules. Except for teachers, very few people think of the rules.
They learn "good" English by imitating "educated" speakers and reading
Best of luck.