It depends what you mean by "grammatical mistake".
Native speakers are less likely to conform to standard grammar when they are speaking. The main reason for this is that, except for very short sentences, we don't formulate the whole sentence before we start to say it. Often, we begin a sentence not knowing how it will end, and by the time we have got to the end of the sentence, we may have forgotten exactly how we started it. When we write, however, we have time to plan our sentences and edit them if necessary, before letting anyone else see them.
It is true that if you stopped somebody in the middle of London and said, "Big Ben is you know where?" you would be understood and nobody would comment. But if you wrote a letter to your supplier and said, "Our order is you know where?" that would make a bad impression.
And also, what is considered a "grammatical mistake" in formal English may be perfectly acceptable in informal English. So we learn the rule that we say "I am" and "Am I?", but "Aren't I?"; but in Scotland it is perfectly acceptable to say, "Amn't I?" -- at least in informal situations. And in many other dialects, in informal situations it is acceptable to say "Ain't I?"
I would definitely say that when you are writing, especially formal letters, you should try to get your grammar right; but when you are speaking, the most important thing is to get your message across, and people will be more tolerant if, for example, your verb doesn't agree with the subject.
But that doesn't mean you never have to worry about grammar. If you were to refer to your female boss as "he", that could be confusing or even slightly offensive.