I said that if it ____ ready by Monday, I would cancel the order.
it being singular will go with "wasn't"...
In every day English, "it was not" is what I would certainly use. In a more literary mode " it were not" could be used.
AS it is reported speech, "wasn't" sounds better to me.
The sentence calls for the subjunctive, so 'weren't' is technically correct, though in spoken speech, it's infrequently used.