Have you ever been to London?
Yes, of course I've been to London.
As for your second sentence, I would use it to say that I had been there doing 'something'.
I've been in London working all week.
I've been in London studying.
Also it answers the question:
How long have you been in London?
Ive been in London for 2 years now.