In the Army, soldiers stand at attention in line. If one of the soldiers is standing out(side) of the line, s/he, to quote RonBee, "is doing something inappropriate, something [s/]he shouldn't be doing."
Yes, but in a different way from 'off the track'.. 'Out of line' means they aren't following rules, instructions, etc. A politician may speak out of line by saying something their party doesn't approve of. 'Off the track' means that the coversation is wandering away from the subject. ;-)