No ____ than fifty people came.
Generally, if you can count it, it is fewer, if you can't, it is less. For example, Less hair/fewer hairs. Less water/fewer drops of water. Therefore in this example, the number of people can be (and is) counted so it is fewer.
I agree that fewer is correct, but for a slightly different reason. Strictly speaking, less applies to singular nouns. Because people is the plural of person, less would be incorrect. The exception in using fewer occurs when count nouns are so great as to render the idea of individual increments meaningless. So less is used correctly with time and money, even if the noun is plural. E.g., "less than two dollars a day."
Fewer, period. As Trolley said. If you disagree, then it's time to go back to school.