The little man, excitable and often tipsy, was terrified of her, and the stranger soon heard of domestic quarrels in which she used her fist and her foot in order to keep him in subjection. She had been known after a night of drunkenness to confine him for twenty–four hours to his own room, and then he could be seen, afraid to leave his prison, talking somewhat pathetically from his veranda to people in the street below.
(W.S. Maugham; The Pool)
The part in bold, in my opinion, refers to 'him'. Is it possible by changing the context to make it refer to 'her'? In other words, does whom this part refers to depend solely on the context?