I would say the book is written in "technical language." Another word is "jargon," but this is used more for speech.
"I tried to use a book from the library to fix my problem, but it was written in such technical language nobody except an engineer could understand it."
"My husband and his friends from work talk completely in scientific jargon. I can't understand a word they say."
Does that help?
[native speaker & writer, not a teacher]
Student or Learner