In this sense and in the negative, no difference. Both are the past negative forms of must: He must have been joking/he can't /couldn't have been joking.
If you have a question about the English language and would like to ask one of our many English teachers and language experts, please click the button below to let us know: