should the direct object always be thing and indirect object a person?
They are building a bridge over the river at the moment.-active voice[ direct object - common noun]
A bridge is being built [by them] at the moment - passive [indirect object - personal pronoun in Acc.]
Grandma looked after the boy while his parents were out dining.-direct object - common noun/person ,as you say
The boy was looked after by grandma when... .-indirect object -common noun/ person
The earthquake caused a lot of damage -direct object- common noun
A lot of damage was caused by the earthquake - ind. obj. - common noun
The bullet hit a window- direct object - common noun/ thing ,as you say
A window was hit by a bullet - indirect object - common noun/thing
Does it help?