Here, the question is about adapting the idiom to a contet- idioms are not cast in stone and we can change and adapt them for specific contexts. If you didn't have the word 'that' (or 'those') in the sentence, you wouldn't need to adapt, but once you refer specifically to the thing left, then it changes the context.
