I was clad in white robes who were slowly turning grey.
Don't you think that would be better?
I would use which if the sentence would be something like:
I was clad in white robes, which was very embarrassing.
(Please do not focus too much on its sense )
In this sentence you don't describe the white robes directly.
That's why I would use which.
In the original sentence, however, I would go for that, because you describe them.
P.S. Not sure about the comma.