Do people sometimes inadvertently use "literally" for "virtually"
and vice-versa? Or maybe I am just confused.
Yes, in fact, people often use literally when they really mean the exact opposite.
"He was so angry that his head literally exploded."
I hear that type of construction all the time.