Less doesn't carry more of a "math notion."
The distinction is made between countable and uncountable items. We have less food. We have fewer apples.
This distinction is not always followed. The first sentence would have to be "It took me 3 fewer hours to drive..."