One of the things wrong with your sentence is that when you say "didn't live that long…" the "that" usually refers back to a length of time that has already been mentioned.
Otherwise, "The people didn't live that long." means that they didn't live particularly long lives, and in both cases the sentence would probably end there.