Can you explain "exclusive circle" to me?

[Joe]

"Let him meet with a young woman once and he would approach her with an air of kindly familiarity... If he visited a department store it was to lounge familiarly over the counter and ask some leading question. In more exclusive circles, on the train or in waiting stations, he went slower. "

I looked up both "exclusive" and "circle" in a couple of dictionaries, but still couldn't figure out what "exclusive circle" means. Can you explain it to me?

[Tdol]

Exclusive circles are social groups that are very difficult to get into, like the rich and famous or the establishment.