If you say "We have to have been" it means that it is a current requirement that we were joking yesterday. If you can think of a situation in which this makes sense, then you can use the "We have to have been" formation.
If you say "We had to have been" it means "We must have been" -- the only thing that makes sense is that we were joking. This doesn't makes much sense to me. Surely the person saying this knows whether he or she was part of a group that was joking.