Don't you think that would be better?
I would use which if the sentence would be something like: I was clad in white robes, which was very embarrassing. (Please do not focus too much on its sense :-D)
In this sentence you don't describe the white robes directly.
That's why I would use which.
In the original sentence, however, I would go for that, because you describe them.