I've met "hard cheese" meaning "tough". It is usually used to indicate a lack of sympathy with a situation.
A: I wanted to go to the cinema but I ran out of time
B: Hard cheese
Can you give the context of "hard potato"? There is a computer programming term "hard potato"