Speaking to a group of people (even a small group) is actually a very common fear or phobia. There are many people who break out in a sweat at the very thought of having to talk to more than one person at a time. And making a speech or presentation? Forget about it! I have a friend, a US-born native English speaker, who took a failing grade in one class rather than participating in the final exam, which involved making a speech in front of the class because she was that terrified of speaking in front of a group.
Whether or not you're a native speaker, or if you have an accent, or are unsure of your grammar, just remind yourself that this is not a "life or death" situtation. If you make a mistake, nothing serious will happen to you. Even if you're just talking to one person, he is concentrating on what you are saying, not how you're saying it. Once you learn how to relax and not worry about your grammar or your accent, your brain will "unfreeze" and your English will flow more smoothly. The secret to becoming a better speaker is practice, so you have to talk to people in order to improve. Just like learning to swim - you can't do it without getting into the water.