I think you're right - this expression is often used to introduce information that the listener probably won't like, or isn't expecting to hear.
However, it's not exclusively negative - it's just that when someone is being perfectly frank (honest), it can surprise people!
Ex: I liked your paper very much, Kim. To be frank, this is the best student essay I have ever read.
Student or Learner