We can say "there's" and stop racking our brains
It's not of any help, but at least it bumps your post.
I changed my mind several times before I decided I would go for the singular. It seems to me 'that' places some emphasis on the number itself. I understand "that numer" as "the number of kids I told you about".