Yes, It's correct!
Now try to distinguish between subject and object of a sentence. If you (yourself or someone is an object) you can use 'whom' after it...
Let's say.
To whom do you (subject) wish to speak?
I (subject) want to speak with a man (object) whom I met yesterday.

