Come with us vs go with us?

[keannu]

I heard that "go" is normally for a third place away from both the speaker and the listener, while "come" is for the listener's place. But considering the following, it doesn't seem to be a fixed, strict rule. What do you think?

ji3-4-4
Police office : You're under arrest.
Kreisler : Under arrest? What for?
Police Officer : I'm sure you stole Kreisler's violin. Come with us to the police station.

[5jj]

'Come' is often used when the destination is a place where the speaker and person addressed are, or will be.

A (Speaking on the phone): Hi Pete, would you like to come over to my place for a meal?

B: (Speaking on the phone): I'm bored; can I come over to see you?

This is common when 'come with' has the idea of 'accompany:

C: Would you like to come (/go) to the cinema with me this evening?