"If her children didn't know she was studying for the test, nobody would know if she failed."
If i) her children didn't know she was studying. AND
If ii) she failed, THEN
Nobody would know [that she failed].
No, you can't use "whether".
The only thing concerning the woman is that, if she failed, her children would know that she had failed.
"if she failed" seems to be playing two roles. It means, "Nobody would know that she failed, if she failed". "If she failed, nobody would know that she failed".
"Whether" only covers one of these meanings: "If she failed, at least nobody would know whether she'd failed or not".
Student or Learner