I think they might have been expecting you to use the word observe instead of watch.
I am from Brazil
I was having a conversation with a group of english people and they asked me: "What are you going to do tomorrow afternoon." I Answered: "I am going to watch an english class." They told me that I couldnīt say "to watch class" because it is wrong but they didnīt give me an explanation.
According to The Oxford Advanced Learnerīs Dictionary the definition of the verb to watch is: to look at somebody or something for a time, paying attention to what happens. According to the definition above I can say to watch class, is the dictionary wrong?
The person watching the class would most likely be somebody making sure they didn't get into trouble.
You could also say you are going to sit in on an English class. That means that you will be present but will not actively participate in it.