Here's why:Labor Day isn't possessed by Labour (incidentally, that Wikipedia article begins by saying 'Labor Day is an American federal holiday observed on the first Monday in September'. Given the date of your post, I imagine you're asking about the European version, known in the UK as 'Labour Day' - in early May).
It's a day marked to celebrate the power/nobility/worthiness/solidarity ... of Labour.
We have Mothers' Day to honor mothers, and Fathers' Day to honor fathers, and Veterans' Day to honor veterans. Yet with Labor Day, which honors those who labor, we have no apostrophe.
:up: Some time in the seventies, when Harold Wilson (Labour) was Prime Minister, there was a political kerfuffle about a change from 'Whit Monday' (a Church occasion and a Bank Holiday) to 'the early Spring Bank Holiday' (the first Monday in May). Opponents of the change argued (foolishly) that this change was a veiled recognition of an occasion that somehow belonged to the Labour Party.
I know people don't find it persuasive, but that's what "that's the idiomatic way to say it" comes down to. There is not a lot of logic in the way we handle these things. Generally, when one noun modifies another, we use the plain noun. Yet it's Veterans' Day. (Even less logically, many official sites have it has "Veteran's Day, as if there were only one.)