I have never been to London. It's true that "never been/have not been" is related to "to be" but in this case "I have never been to" actually equates to "I have never travelled to".
Q - What have you been doing?
A - I've been to the shops.
Q - Where have you been [on holiday] this year?
A - I've been to Portugal and Italy.