The phrase comes from sea battles. The national flags carried by warships were [and still are] called their "colours", and these were attached to a high mast, from where the colours were said to "fly". When a ship surrendered in battle, it lowered its colours, so a victorious ship would come away from the battle with its colours still flying.