Yes, it's to distinguish New York City from New York State (or even New York County).
In the case of New York, the actual name of the city does not contain the word "city," but people use it to avoid confusion. You can talk about New York and do not have to always add "city."
In other cases, like Oklahoma City, the word "city" is part of the name. You would not just say "Oklahoma," to talk about that city. Carson City in Nevada is another example of cities that formally have "city" in their names.
(Interestingly enough, Mexicans refer to what we call "Mexico City" as just "Mexico." So you can be in the country of Mexico and see road signs directing you how to get to "Mexico.")