It does not have a negative sense, other than the fact that you are evaluating a woman on the basis of her looks alone. You shouldn't really say it to any woman as a way to make conversation.
However, if you know the person well, and she's saying how unattractive she feels these days and you said something like "Mary! What are you talking about! You're a total fox! It takes me three hours in front of the mirror to look as good as you do roling out of bed!" it would be an appropriate compliment.