If you "take it out on someone" it means you subject them to treatment that they probably don't deserve just because you are upset or angry or in a bad mood.
I had a really bad day at work and I came home and took it out on my husband = My day at work was bad, it put me in a bad mood and when I came home I probably shouted at my husband or was very terse and abrupt with him, maybe banged around in the kitchen, rudely said "I suppose I'm cooking dinner again, am I?" and answered his questions with grumpy mono-syllables! Not that I have ever done that, of course! ;-)
It can be worse than just verbal grumpiness to someone who doesn't deserve it. In many cases of abuse, you hear that the victim was always worried that the abuser would be upset or in a bad mood or drunk (or any other negative feeling) and would then "come home and take it [all] out on me". That might have resulted in physical abuse (punching, kicking etc) or even sexual abuse.
Without knowing exactly what the writer's brother did, it's impossible to know what the actual outcome was, other than that the brother vented his frustrations in some way on the writer.
