1. He had been to the barn. He was there at one time in the past, but he is not there at the (past) time referenced. I could tell he had been to the barn by the smell of his clothes.
2. He had gone to the barn.
a. It might mean the same as #1. He had gone to the barn and returned to the house by the time I got home.
b. It might mean he was still there: I asked Mother where Tom was. She said he had gone to the barn.