"Could borrow" would be used for something habitual in the past.
My father never minded if I used his car. I could borrow it whenever I wanted as long as I fillled the tank before returning it.
It's also used for something in the future.
You can't use my car today but you could borrow it tomorrow.
"Could have borrowed" refers to something that could have happened in the past but did not.
So, which one is right?