Good question. These expressions are more or less interchangeable. Both can be used to say that the speaker didn't understand the meaning of what was just said. This is different from the questions "what?" or "huh?" because those can mean "I couldn't decode which words you just used", whereas "What do you mean" and "how do you mean" can only mean "I didn't understand the meaning of the words you just used".
Here are some examples from the Corpus of Contemporary American English:
Joel I don' t even know where to start with this one other than I guess maybe with an interview that I did with Benjamin Netanyahu about two years ago and I said, " You guys are being set up. " And he said, "What do you mean?" I said, " I think the world is going to start pushing towards, Oh, let' s just be friends with Iran.'— from "The Glen Beck Show"
Here, Netanyahu says "what do you mean?" because he doesn't understand what he's being set up for. The other person then explains what he meant by "being set up".
I think you really have to be a Texan to appreciate the vastness of it and the emptiness of it, " Dylan says. " But I'm an honorary Texan. "What do you mean?" I ask. " Well, " he says, " George Bush, when he was governor, gave me a proclamation that says I'm an honorary Texan holds hand up in pledge, toughs. As if anybody needed proof. It's no small thing. I take it as a high honor. "— from "Bob Dylan's America", Rolling Stone
Here, the journalist doesn't understand under what pretense Bob Dylan thinks of himself as an honorary Texan. Dylan responds by explaining that George Bush, as governor of Texas, made a proclamation that Bob Dylan is an honorary Texan.
NELLY: Honest people don't hide their deeds.— from Scholastic Scope
ISABELLA: What do you mean?
NELLY: How has he been living? How did he get rich? Why is he really staying at Wuthering Heights when he hates Hindley? No one knows his purpose.
ISABELLA: I will not listen to you slander him.
Here, Nelly makes a generalized claim about honest people, and Isabella says "What do you mean?" because she doesn't understand how that claim applies to the current discussion's context. Nelly then tries to explain why she doesn't think the person she is talking about is an honest person.
CHARLAYNE HUNTER-GAULT, CNN CORRESPONDENT:What's amazing about these creatures is, they are almost like a world unto themselves.The CNN correspondent says that some creatures are "like a world unto themselves". Ms. Healy doesn't understand in what manner this is true, so she asks "How do you mean?". The CNN correspondent then elaborates that the creatures have algae growing in their fur.
Ms. SUSAN HEALY How do you mean?
CHARLAYNE HUNTER-GAULT, CHARLAYNE HUNTER-GAULT CNN CORRESPONDENT (voice-over) Well, they actually have an algae, a type of plant that grows in their fur.
— from CNN
ROBIN TUNNEY Thank you much. I do appreciate the fact that these vaccines are being taken care of, but one thing I have a problem with, living in California, is identification of people that have something like that.Senator Boxer doesn't understand how the caller expects to identify the people in question.
Senator BARBARA BOXER (Democrat, California) How do you mean? You mean finding out if somebody has been vaccinated or somebody has smallpox?
ROBIN TUNNEY Well, if people are being tested for this stuff.
Mr. NAESSENS Well, but, I say, if we can go together, we would go together.Mr. Robinette doesn't know what Mr. Naessens means by "go together", so Mr. Naessens explains that he means "die at the same time".
Mr. GARLAND ROBINETTE How do you mean?
Mr. NAESSENS Well, die at the same time.
I have two intuitions about these phrases, but no evidence to back them up:
In my opinion, in all of these examples, the other phrase could have been substituted equally well. I had originally thought I might be able to find some nuance of meaning from my intuition that differs between the two but I was unable to find any examples where the opposite phrase wouldn't have worked as well.
- First is that the difference between "what do you mean" and "how do you mean" is degree of confusion; "what do you mean" indicates more confusion than "how do you mean".
- Second is "how do you mean" is asked more often when the speaker is requesting "how" information, meaning "in what way?" or "in what manner?"
( From Nohat)