You're right: it's not correct.So, "Yes you are speaking to" is not correct.
If you have a question about the English language and would like to ask one of our many English teachers and language experts, please click the button below to let us know: