"Holding back the work" might mean not providing work that needs to be done.
Student or Learner
The others quite rightly object that he is holding back the work. This sentence from COLLINS COBUILD ENGLISH USAGE is translated in the Chinese version as 'The others strongly object that he is concealing the work'.
But I would think this sentence means 'The others are right in objecting that he is hindering the work from progressing. Am I right?
I do agree with your interpretation the others are right in objecting- strongly doesn't work for me.