Singular: That chair that has a cushion is comfortable.
Plural: Those chairs that have cushions are confortable.
However the word "that" in the sentence you gave is an introductory word to introduce the relative clause "have cushions" if it's the "that" you referred to. In this case, it indeed doesn't have any other form when serving as an introductory word for clauses.