I thought about that, but chickened out of mentioning it because it doesn't seem to fit in the scheme:
very few < few < a few < quite a few
While it doesn't seem to fit in, I think it does - so long as you think of the "very" as intensifying the word following it. This means that 'a very few' is less than 'a few', but more - I think - than 'few'. When saying 'a very few', the speaker has decided they want to say something more than 'few', says 'a', and then realizes there's a risk of exaggeration - so adds 'very' to tone down 'few'.
This makes the full sequence:
very few < few < a very few < a few < quite a few
But as with all such clines, theory is neater than practice and views differ.