***** NOT A TEACHER *****
I do not know whether one can use "can't go" in that
sentence, but I think that most native speakers would be
more comfortable with "can't come."
Let's see whether I can explain the reason.
(1) Mary and Paul are on the phone. Mary is at home.
Paul is at work. Mary is calling Paul. So when Paul talks
with Mary, he is in (mentally speaking) already at her
house!!! So when he says, "I can't come," he is "mentally"
saying, "I can't come here [Mary's house]."
(a) Here is another example that may make my explanation
a little clearer.
Mary (at home) is talking on the phone to Joe, her husband at work:
Mary: We don't have any bread.
Joe: Don't worry. I'll bring some bread home tonight.
If Joe had said, "I'll take some bread," that would have sounded
strange. When Joe was speaking with his wife, he was "mentally"
already at her place (home). So you always bring something here,
but take something there.
By the way -- returning to your question --consider this telephone conversation:
Mary (at home): Don't forget to visit Mona in the hospital.
Paul (still at work): I have to work late, so I can't go to the
hospital today. It will have to wait until tomorrow. But don't worry.
I will definitely come to your party tonight -- although I will arrive
THANK YOU & HAPPY NEW YEAR
Student or Learner