The answer is that you are doing more than just omitting the relative pronoun -- you're omitting a verb as well. Consider:
The woman (who) I wanted to see is on holiday.
If you omit the relative pronoun, you end up with a sentence that makes sense: "The woman I wanted to see is on holiday."
"Who" replaces "the woman" which is the object of the sentence "I wanted to see the woman".
Now take one of your examples:
The man (who) is wearing the glasses looked at me.
If you omit only the relative pronoun, you end up with:
*The man is wearing the glasses looked at me.
"Who" replaces "the man" which is the subject of the sentence "The man is wearing the glasses", and the rule holds good.
You are, in fact, changing the structure of the sentence. Whereas before you had a relative clause, now you only have a phrase -- with no finite verb. That's a completely different animal.