You would have preferred to have joined me though.
You would have preferred to have had joined me though.
Ask yourself: What was the preference? -> to join him.
'You would have preferred to join me though' is quite possible - in the appropriate context.
PS (for the non-fence-sitters): I know there's a tendency to reduplicate perfects meaninglessly in sentences like this; I can't for the life of me think why.
I wonder if this is another one of those American differences. The versions with "have" in the second part sound very natural to me.
I'm not a teacher, but I write for a living. Please don't ask me about 2nd conditionals, but I'm a safe bet for what reads well in (American) English.
I would use either:
I would have preferred to join him.
I would prefer to have joined him.
But they mean different things. Each is right in its place. Sometimes one's preference at some time in the past may differ from what - now - one would have preferred.
'I would have preferred to keep my cool, but I'm afraid I lost my temper.'
'I would prefer to have kept my cool; it would have been more adult.'
The distinction is slight, but sometimes I think it's worth making.