The worst thing about teaching could be different. It depends on the situation.
My worst situation in my case was being mandated to each so students could pass a test. I felt I should teach them to be able to succeed in life by knowing not just "English" but Language Arts, including the ability to comprehend, imagine, read, write understandably, and to think for them selves.
Your supervisor can be your worst night mare if they have predetermined ways they want you to teach. My first job teaching was to replace an English teacher who was promoted to Principal of the school. He was older than dirt and expected students to memorize everything. Comprehending and using that information was NOT important, only to "know it".
Salary can be a nightmare if you can't support yourself and family. Working two jobs so you can afford to teach takes the joy out of teaching.
Students may be assigned to you and you are totally unable to teach them because of problems for which you have no control. If you get a "slightly retarded" student, you will never get them to comprehend the same level of an "average student". This is especially fustrating when both students are put in the same room and you have to teach them along with 30 or 40 others.
I always looked for that one student that I could find and was able to provide them with knowledge, wisdom, and understanding. It made the day and school year worthwhile and brighter.