I don't think that there is such a rule.
"He was heard to mention your name several times.
They were seen to leave the house early in the morning."
In American English the above sentences would sound unusual. Here is how we would say it:
He was heard mentioning your name several times.
They were seen leaving the house early in the morning.
It seems to me that the use of to with the infinitive is often idiomatic.
"I will ask him to call you."
"I will have him call you."
Student or Learner