I tried ____ her but the line was busy.
The statement is, in itself confusing because you were, at this particular moment in time performing the action of 'phoning' someone. Surely, it would be better to say "I tried to speak to her . . ", or "I tried speaking to her . . ". Or, maybe even "I tried to contact her . . " ?
I think it maybe "I tried phoning her" as the sentence is in the past, but am not sure.
I tried to phone her . Because it is smart to use infinitive here.
phoning - if you are able to make a number of attempts.
to phone - if there's an impediment to accomplish the action.
It is to call not to phone