Imagine you're swinging something round your head. It's moving in a circle. Let go, and it flies off at a tangent. So, figuratively, a discussion or argument flies off at a tangent when it suddenly insists on following one (irrelevant) direction.
(Sometimes it's used in a slightly different sense - moving away violently from one direction. But anyone who's aware of the root [tangere - Latin "to touch": a tangent is a straight line that touches a circle] might be tempted to smile inwardly at what classics teachers might regard as a "mistake"!)