I am not a teacher, but I think both sound natural, only with different meanings.
It's too heavy for me. (I am sure about it).
"I only put the rubbish out once a month because the pods are too heavy for me."
It looks too heavy to me. (That's my appreciation anyway).
"A racquet that seems too heavy to me, may seem light to someone else."
That's only my personal point of view.