What are "learn grammarians"? Did you mean to write "learned"?
I totally agree with you that determining whether a clause is restrictive or not can be tricky. This is mainly because it's difficult to define the term "restrictive clause." Supposedly, it denotes information that is essential to the meaning of a sentence, but I find that many times such information is nevertheless set off with commas, which is typical of nonrestrictive clauses. Actually, I myself have started many threads on this forum asking whether a clause was restrictive or not.
I don't quite understand the example you provided, but if you're asking if there should be a comma after "time," the answer is "no."