But "upstairs" is not a "place." It is a relative relationship.
The 10th floor in Building A was "upstairs" to people who live on the lower floors, and "downstairs" to people who live on higher floors.
Ex: Joanne lives on the 15th floor, so she goes downstairs to visit her friends on the 10th floor. But Patty lives on the 8th floor, so she goes upstairs when she visits them.
So the person who moved to the 5th floor in Building B is still "above street level" -- which is one way to think about "upstairs." But he's downstairs according to his friends on the 12th floor.