Yesterday evening...
Last evening...
Are both correct?
My personal view is that 'yesterday evening' is far more commonly heard/used than 'last evening' in most dialects of BrE.
'Yesterday night', however, is, in my opinion, unacceptable. We can say only 'last night'.