"Yet" is used in negative sentences but your example was a question. Your question was neutral - it could attract either a positive or a negative answer.
Have you spoken to him yet?
Yes, I have. I spoke to him two hours ago.
Have you spoken to him yet?
No, I haven't spoken to him yet. I'll probably talk to him this afternoon.
As you can see, the positive answer in the first example does not use "yet" but the second answer, which is a negative sentence, uses "yet".