Singular if you're talking about the family, plural if your emphasis is on the birds.
If you're unsure, think about what will come after that sentence:
If you use the singular - the family...nests - then further references to it must also be singular. Eg: 'It then leaves early in the autumn.'
If you use the plural - the family...nest - then the next sentence might read: 'They make a terrible noise all summer.'
Avoid mixing it up, as 'A family of mockingbirds nest in the tree and wakes us up every day' would be a serious grammatical crime.
Most 'nouns of multitude' like this one (army, hunt, crowd, etc) behave the same way. There will be exceptions, I'm sure.
Student or Learner