I didn't know whom was ever used with an adverb in front of it. It is used instead of "who" as the object of a verb or preposition:
-object of preposition
-object of verb (the person that receives an action, like him, her or us)
whom are you going to tell?
whom did he kill?