IMO, swimming is not a predicate adjective.
The word "supposed" is an interesting one. What part of speech is it? In my opinion, it is part of a passive voice construction in the present tense. One supposes things, concepts, and even people. In this case "We are supposed" means that others are supposing us to be doing something. It has taken on the meaning of obligation. The verb suppose takes an infinitive. We are supposed to be skiing. "To be skiing" is the progressive infinitve of "to ski" -- "to be skiing".
I am skiing.
You are skiing.
He/she/it is skiing.
In the "to go skiing" form, either "to go" is an infinitive and skiing is a gerund object of the infinitive, or "to go skiing" is a phrasal infinitive with "go" acting as a helping verb. I prefer the second explanation, but most dictionaries give that use of "go" as a transitive use, requiring a direct object. I think either explanation is acceptable.
That's my take on this.