I think he means why don't we just say I went to the doctor, the butcher, the candlestick maker....
And I guess the answer is it's more polite to describe visiting the place belonging to the person than the person himself.
Cf. in French: we say chez le médecin, chez le boucher.... also a form of possessive in a way.