If it is said to a group, "books" would be used. If it is used for an individual, "book" would be fine.
what's the grammatical rule for " open your books"
why can't we say " open your book"
Look at my corrections above. It's important to use correct capitalisation and punctuation in written English.
The simple answer is that you need to consider how many books you are referring to. If there is one person with one book, say "Open your book". If there are several people with one book each, or more books each, say "Open your books" (you want every book to be opened). If there is one person with three books, say "Open your books" if you want the person to open all three books.
Remember - if you don't use correct capitalisation, punctuation and spacing, anything you write will be incorrect.