NOT A TEACHER
I have heard native speakers say things such as, 'You idiot', so, yes, I guess it's pretty common in spoken English. There is an implied verb, 'You (are an) idiot'.
You cannot, however, say, 'He/she idiot' (although I can't say why). You also cannot say, 'You smart'. 'You genius' does work on the other hand.