Does "To be frank with you" mostly mean a negative nuance like in the below? Don't they use this for positive ones?

ex)To be frank with you, that sweater doesn't look good on you.