I (subject) want to speak with a man (object) whom I met yesterday.
I would probably use "who" istead of "whom" at above sentence!
So is this wrong? --> "I want to speak with a man who I met yesterday!"
It's fine and what many native speakers would say, though man I met and man that I met are also used. The traditional subject/object rule for who/whom is fine if you want to use it, but most native speakers do not. In very formal language, you might want to use whom. About the only time I would say you have to use whom is directly after a preposition- To whom it may concern, etc.