The meaning of the word "downtown" varies with the area. In Chicago, "downtown means the central commercial area, as Coffa indicated. One goes downtown no matter which direction one is traveling to get there. In Manhattan, New York, "downtown" means south or towards Brooklyn or Staten Island. "Uptown" means north or towards Queens or the Bronx. "Crosstown" means east or west.
In addition "downtown" can mean shabby and economically deprived, and "uptown" can mean just the opposite. This is interesting because one of the most depressed areas in Manhattan (Harlem) is north and one of the most expensive areas (Greenwich Village) is south. Go figure.