If it would've had an answer key, I think the book could've been more helpful.
The sentence doesn't make sense, whether speaking or writing! Completely unacceptable.
If it had had an aswer key...
Had it had an answer key...
I would just like to know if it would have had an aswer key.
First off, I really don't like using two similar contractions in the same sentence, it sounds weird. But is the sentence wrong because it mixes would and could? It would sound Way bettter if "would've" was removed from the first part.
The sentence is basically unacceptable. That said, any native speaker would have little difficulty in interepreting it. The two problems are the verbs: "would've (would have) had" is needlessly phrased as a conditional when the clause should just have been 'If it "had had" an answer key'; and "could've" ("could have") should be: 'the book "would have" been more helpful'