We use it a lot.
Are you coming to me or am I coming to you?
It doesn't have to mean "my/your house" in all contexts. For example, I am going out for dinner with a friend next week. She will be going from her house and I will be going straight from work. However, we have agreed to catch the bus into town together. Before going to the bus stop, either I will go to her house or she will come to my workplace so that we can walk to the bus stop together. If I say "Are you coming to me or am I coming to you?" in that context, it would mean "Is the plan for me to come to your house or for you to come to my workplace [before we go to catch the bus]?"