"Otherwise" acts as an adverb to modify the verb "do".
1. ...so long as they considered that to do otherwise would be unfairto the defendant.
2. ..so long as they considered that to do so would be fair to the defendant.
IMO, it is not reasonable to derive the second clause from the first.
not a teacher