The first sentence is correct:
1. Did you see him doing that?
In this sentence, "him" is the object of the verb (i.e. the thing that is seen), and "doing that" is a participle phrase which gives more information about "him": you didn't just see "him", you saw what he was doing too.
The second sentence on the other hand is incorrect, as can be seen if we remove the participle phrase:
2. Did you see he?