Almost always, they are interchangeable.
If the context concerned saving money by reducing expenditure I would probably prefer most inexpensive. But believe it or not, there are people in this world who think that more expensive is always better, and much more expensive is always much better. When the context concerns such people, more expensive is probably better.
To illustrate this point, I once talked to a young man who was unhappy that he had lost his brand new $1,000 pair of sunglasses. When I asked him why anyone would spend $1,000 on a pair of sunglasses in the first place he explained that his friends shopped in the same stores as he did, and when they saw those sunglasses they would know that he could afford to waste $1,000 on a pair of sunglasses. To such people, the word inexpensive is contemptible, so one would probably prefer least expensive when talking to them.