Your instinct here is correct. The "used" here is the hypothetical subjunctive, because the condition of people using violence against him is not necessarily a given. It's similar to saying "even if others were to use violence against him". It indicates the hypothetical potential for violence, rather than past habitual violence.
Student or Learner