One can go to the college or the school and teach, or visit one's children, or take a class, or just turn around and come home. All you're saying is that you're physically traveling to the location in question. To go to college/school is to attend classes and/or embark on a course of study.
One can go to the school without going to school, but it's hard to go to school without going to the school, if you know what I mean.
The definite article is used:
1. to refer to something which has already been mentioned.
2. when both the speaker and listener know what is being talked about, even if it has not been mentioned before.
3. in sentences or clauses where we define or identify a particular person or object.