The rule is simple at first sight: we consider "church" as an activity, while "office" is a building or a room.
That seems simple, but why is "church" an activity and "office" isn't? The answer is: it's in the mind of the native speaker.
We can differentiate:
"I am going to church" means that I am going to participate in worship (including all the hymn-singing and praying); "I am going to the church" means that I am going to a specific building -- I may not be going there to worship, I could just have arranged to meet somebody there, for example.
The same is true of "school". You go to school to learn, but you go to the school to speak with the headmaster, or to bring your children home.
But we don't consider an office to be an activity in the same way. However, what we do in an office is (normally) work, and that's the activity: "I go to the office" or "I go to work".