"Gripe my liver" is not at all common usage (or even common slang), but what he means is that this type of girl really upsets him, gets on his nerves, makes him roll his eyes in exasperation.
"I never thought I'd come to the point where that would be an issue."
There is a common expression among the wealthy/elite: "If you have to ask how much it costs, then you can't afford it." So in this piece of dialog the speaker is suddenly having to budget his money; he/she cannot just purchase any item that catches his fancy. He now has to ask how much it costs and then decide if he can really afford to spend that much money.