Yes, it depends on context. But in general, as I know, 'to' is connected with motion or direction,it is used more after verbs ('COME to me'). 'For me' has the different meaning - it is 'a present for ME', ' it's not interesting for ME', so 'for' refers to an object.