There isn't much difference. The use of "for" might mean in some contexts that you left, got your lunch and returned to eat it. Whereas, going "to" lunch might mean you went somewhere and had lunch there..
Just be careful about saying this
He is out to lunch.
This means that he is thought to be crazy or weird
Student or Learner