It refers to a definitive piece of evidence used to prove someone's guilt. Literally, it means a gun that has been fired recently, and therefore would actually have smoke wafting from the barrel or the aroma of smoke if an investigator sniffed at the barrel end. A figurative example is the Watergate scandal involving President Richard Nixon. Nixon denied any involvement in the matter, but a tape recording was eventually discovered with Nixon's voice on it ordering his staff to halt the FBI investigation into the Watergate issue. This tape recording became the "smoking gun" - the vital piece of non-refutable evidence.