The lady to whom I spoke on the phone.
or The lady whom I spoke to on the phone.
Most people will probably prefer the first, but would probably also agree that both are acceptable. You might like saying "with whom" rather than "to whom," since phone conversations are usually two-way. However, "to whom" is fine.