In truth, I'd say "There aren't very many of us."
We are few in number, but mighty in determination.
There are only a few of us (emphasizes not many).
There are a few of us (some with no sense of whether that's good or bad)
There are few of us (emphasizes the smallness of the number)