And typically, with English, it doesn't always literally mean to go down in height, or to head South. Sometimes people just use it instead of saying "go".
Q - Where are you going?
A - I'm going down town. (Same as "I'm going into the local town")
In a famous children's song "The Teddy Bear's Picnic", the first line is "If you go down to the woods today". There is no suggestion that the woods (forest) are necessarily to the south of where you start, or that they are downhill.
up and down can usually both be used with the same meaning.
For position though 'DOWN town' usually means the city centre. Even this is not a hard and fast rule and so 'up town' can also mean this if you say you are going 'up town'. It depends on the context. In some areas 'up town' can mean the suburbs outside the city centre.
You can be travelling in the same direction as your friend and he can be going 'up the road' but you might say that you are going 'down the road'. If you look 'up the street' and he looks 'down the street' you may both be looking in the same direction.