Sorry, there is no easy answer. It seems that the usage is by custom or idiom. In the back of the room and at the back of the room mean exactly the same thing. In the back of the car means in the back seat or rear compartment of a car. At the back of the car means outside of the car, near the rear of it.
There are probably hundreds of examples of each type of usage; these are the ones I thought of off the top of my head.
If you use in every time you describe someone or something inside of something, and use at when someone or something is outside of something, you won't go wrong.
BE usage agrees, as far as that general principle goes. But there is a peculiarly AmE usage of 'in back of' (no the), where BE prefers 'behind'. Of course, as with much (originally) AmE usage, 'in back of' is sometimes heard in BE, but in most contexts it's not widely regarded as acceptable BE.
The 'in back of'/'behind' equivalence works only for actual/concrete 'behind': 'in back of the building' =(arguably, in different dialects) 'behind the building'. But I don't think 'in back of' works in a figurative context: He knew who was behind the killings, but he had no proof.