I think you explain things very well.

This is my theory: When we say "something is right out of town" or "a long way out of" or "the hotel is out of town", we are thinking about a trajectory and a distance to be covered by normal, everyday means. We think about a line from point A to point B and the fact that that line crosses a boundary. In other words we are not really thinking about locations per se, but about a movement and a boundary. You can't think like that when it comes to New York and Europe (there is no boundary in the real sense of the term.) You won't say a hotel in LA is out of town if you were in New York and if you said 'It is a long way out of town.' you'd probably be using irony. Am I right?

When we say "He is out of the building." don't we imply that at one point in time he was in and has now gone out? Movement again.
When we think only about location, we use outside. It is geography we are talking about.

This is only a theory, but what do you think about it? I think it takes into account everything you have said on the subject.