I tried ____ to her, but she wasn't persuaded and refused.
I think it's " to talk" the right answer.
Surely 'talking' and 'to talk' are both acceptable
"Talking" and "to talk" are both good to use.
I'd say 'talking' in this case. There is a difference between 'tried talking' (sounds as if you talked, but couldn't come to an agreement) and 'tried to talk' (you wanted to talk to her, but she avoided/blanked you).
'Try to talk' means you're doing it over an impediment, you're hardly able to; 'try talking' means you're making a try to see a consequence of your action.