"In her company" means that you are accompanying her, you are with her as a companion.
In this case, the man is worried about being seen out in the company of this woman (who isn't his wife) because it might give the wrong impression. It may look like he's not just her companion, but like he has a romantic relationship with her.
