Start with this:
Find the prime factors which yield 864.
This can stand alone (which means that 'when multiplied by each other' merely gives extra information). It also means that 'which yield 864' is a defining relative clause. There is no pause between 'factors' and 'which', and 'that' can replace 'which'.
So, there is no comma between 'factors' and 'which'.
We have seen that the 'when' clause gives extra information. This information is bracketed off by slight pauses in speech; we can do that in writing with commas. SO:.
2. Find the prime factors which, when multiplied by each other, yield 864.