(Not a teacher)
'Come to me' is a request for the listener to move towards to speaker. The speaker is probably not moving, nor going to move.
'Come with me' is a request for the listener to move with the speaker. The speaker is moving, or will move, and wants the listener to move together with him/her.