"Would you like to have lunch with me?" is fine. You might want to preface it with a question, such as "Do you have any plans for lunch today?" If they reply "No" then you can extend the invitation. If they have a previous engagement, you can re-word your invitation to "Would you like to have lunch with me sometime this week?" or something similar.
