"Looked at" is not as good as "watched" -- which is not an option.
"Listened to" is perfectly grammatical. Perhaps they are blind. (There was a character on a TV show who was blind but who loved basketball so much he could picture the action based on what he heard.) I also remember many hours spent in my room, which was over the garage and the basketball hoop was attached over the garage door. I certainly heard a lot of basketball playing growing up. I didn't usually actively listen, but I could have.