Yes, "was driven out" is the past simple passive.
They drove me out. (Past simple active)
I was driven out by them. (Past simple passive)
They have driven me out. (Present perfect active)
I have been driven out by them. (Present perfect passive)
They had driven me out. (Past perfect active)
I had been driven out by them. (Past perfect passive)
Student or Learner