Thank you very much.
However, that doesn't work there in my opinion. Up to this chapter there isn't mentioned anything about postage stamps or stamp collection. Even if stamps or a stamp collection became a subject of the plot it would be meaningless or nonsensical to refer to at this point. Maybe the source you have consulted doesn't actually apply.
He is talking about the following (to make things clearer):
(Loudon Dodd talking about Pinkerton. In the OP Pinkerton is speaking)
It was a subject often and earnestly debated by myself and Pinkerton. In his opinion I should instantly discard my profession. "Just drop it, here and now," he would say. "Come back home with me, and let's throw our whole soul into business. I have the capital; you bring the culture. Dodd and Pinkerton - I never saw a better name for an advertisement. . . " I told him, besides, that however poor my chances were in sculpture
(Loudon Dodd is a sculptor), I was convinced they were yet worse in business, for which I equally lacked taste and aptitude. But upon this head he was my father over again; assured me that I spoke in ignorance; that any intelligent and cultured person was bound to succeed; that I must, besides, have inherited some of my father's fitness; and, at any rate, that I had been regularly trained for that career in the commercial college.
(Loudon Dodd attended a school named Muskegon Commercial Academy where he was trained to become a perfect businessman but what he wasn't interested in at all. In this Academy a pseudo exchange was integrated where the students could experience the clamor and excitement of a real exchange)
"Pinkerton," I said, "can't you understand that, as long as I was there
(in the Muskegon Commercial Academy), I never took the smallest interest in any stricken thing? The whole affair was poison to me."
"It's not possible," he would cry; " It can't be; you couldn't live in the midst of it and not feel the charm
(. . .of the pseudo exchange 
); with all your poetry of soul, you couldn't help!
And then Pinkerton is saying (what has been said in the OP):
"Loudon, you drive me crazy. You expect a man to be all broken up about the sunset, and not to care a dime for a place where fortunes are fought for and made and lost all day; or for a career that consists in studying up life till you have it at your finger-ends, spying out every cranny where you can get your hand in and a dollar out, and standing there in the midst - one foot on bankruptcy, the other on a borrowed dollar, and the whole thing spinning round you like a mill -
raking in the stamps, in spite of fate and fortune."
. . . and then Loudon Dodd goes on:
To this romance of dickering I would reply with the romance (which is also a virtue) of art; reminding him of those examples of constancy through many tribulations, with which the
rôle (italicized in the original, a French word) of Apollo is illustrated - from the case of Millet
(a French painter), to those of many of our friends and comrades, who had chosen this agreeable mountain path through life, and were now bravely clambering among rocks and brambles, penniless and hopeful.
