I think "I had gone there" would not necessarily mean you were still there.
"I wasn't sad when the restaurant went out of business; I had gone there for dinner a few times, but the food was mediocre. I hoped the new owners would do a better job."
While "been" and "gone" can sometimes be used almost interchangeably, "had been" describes the state of being there, while "had gone" refers to the journey there. Thus, "been there" can cover more time.
"I had been there for two years when the company went bankrupt."
"I had gone there to take a new job. Two years later, the company went bankrupt."
But: "Have you been to the new nightclub yet?" and "Have you gone to the new nightclub yet?" mean just about the same thing.