I think it's a litle more complex than that.
If the speaker an/.or listener are in the listener's office at the moment of speaking, then the speaker is likely to say 'came'.
If neither is in the listener's office, thn the speaker is likely to say 'went'.
However, there is no 'rule' about this - it's a matter of personal perception. So, if I mentally associate the listener with always being in their office, I might well use 'came' even if neither of us is there at the moment of speaking. This explains why we sometimes hear people use the word we would not normally expect.