If someone is talking out of their hat, they're talking utter rubbish, especially if compounded with total ignorance of the subject on which they are pontificating.
('Talk through your hat' is also used.)
A speaker says "that makes two of us" to indicate agreement with what another speaker just said. For example, I can say, "I wish I would win the lottery." A listener who says "That makes two of us" is indicating the he or she wants to win the lottery, too.