Well, all I can say is that 'when' and 'where' are relative adverbs, not relative pronouns.
When the relative pronoun functions as the object in the relative clause, it can be omitted.
"This is the shirt (that) I'm going to wear." - 'that' functions as the object in the relative clause.
If you split the sentence into two, it will be easier to understand:
"This is the shirt." "I'm going to wear it."
I hope you'll understand what I mean.