Because the word straight has so many meanings I think the first sentence has at least two meanings which immediately come to mind:
1. a) He went straight up the hill = as you said, by the quickest or most direct route possible, and also if you think about it the quickest means for him to get up the hill would be in a straight line. So, if he was on foot, say, he could chose to walk in a straight line up an open hillside. However, without knowing the scenario you can only assume it's by the quickest means possible, even if that route has bends in it.
1. b) He went straight up the hill = he went up the hill immediately (i.e. at once). Normally in BrE we would probably say "He went up the hill straight away" for the sense of doing it immediately. I'm not sure if AmE uses "straight away".
2. Doesn't quite work for me. I would more naturally say something like "He looked straight up", as in "He looked straight up at the sky above him". So here, as you thought it means in a direct, or straight, line above him. I don't get the sense of doing it immediately though. I think that you get that sense more when someone's moving, or going, somewhere. e.g. He went straight to the shops.