Davy, why can't I say "last afternoon" or "Last morning?"
My own view on this question is that the pattern arises through usage changes occurring sometime between Early Modern English (1500-1700) and Late Modern English.
In EME, it would have been correct to say 'last morningtide' or 'last afternoon', just as it would have been correct to say 'at morningtide' or 'at afternoon'. For example, "I have been with the ague since this last afternoon" is perfectly correct in EME, but (though understandable) sounds naturally archaic to a native speaker of LME, who would say "I've had a cold since yesterday afternoon."
In today's LME, we do not use the word 'morningtide' and we do not say 'at morning' or 'at afternoon' - we say 'IN the morning' and 'IN the afternoon' instead. The use of the adjective 'last', with the meaning of 'previous', is tied to this usage change, which has not occurred uniformly. We do still say 'at night', so we do still say 'last night' (just as we say 'at Christmas' and 'last Christmas'), but we do not say 'last afternoon' or 'last morning'.
Having said this, I do not agree with Davy that 'last evening' is correct either in current usage. I certainly wouldn't say it, anymore than I would say 'at evening' for the same reason - and I don't know anyone else who would. The only 'correct' usages are 'yesterday evening' and 'the previous evening'.
I should point out that when I say 'correct', I am not saying that this is a rule - merely that is is standard usage in the English spoken today by native English speakers. You WILL hear people say 'last evening' (and 'last afternoon' too sometimes), but they are not conforming to the current norm.