You're probably trying too hard with the native speakers and don't feel the same pressure to perform when speaking with non-native speakers. If you could learn to speak to native speakers with the same focus on communication and not on trying to be perfect, you'll probably speak better to them too. It's not an uncommon issue. Try relaxing more around native speakers, most of whom will be more interested in what you're saying than how you're doing it. Always remember that you're doing the hard work when you're speaking a second language.
Student or Learner