give up : cease making an effort; resign oneself to failure.
If you are trying to solve a difficult problem, but can't, then you might say, "Oh, I give up." Also applies to trying to put together Ikea furniture!
Or the person asking a riddle might say to the other, "Do you give up?"
give in : cease fighting or arguing; yield; surrender
Two people are having a wrestling match, and John has Paul in some hold that he can't break out of. John might then ask,"Do you give in?" That is, 'surrender'.
Recognizing in oneself that you can't 'win' or 'do it' - that the situation, whether a battle or a riddle, has defeated you - will then cause you to 'give up' or 'give in'
Any clearer or not?