thanks for all explanation.
And typically, with English, it doesn't always literally mean to go down in height, or to head South. Sometimes people just use it instead of saying "go".
Q - Where are you going?
A - I'm going down town. (Same as "I'm going into the local town")
In a famous children's song "The Teddy Bear's Picnic", the first line is "If you go down to the woods today". There is no suggestion that the woods (forest) are necessarily to the south of where you start, or that they are downhill.