'Spoiled' is not natural to me. I'd say "It is sour" or "It has gone off".
Context is important. Please provide enough for us to be able to deal effectively with your question.
Your thread title should include all or part of the word/phrase being discussed.
If you just want to know the meaning of a word, try OneLook Dictionary Search first.
About the main question, it depends which milk is actually being spoken of. If by "the milk we had for breakfast", you mean the container of milk from which we poured our own," you can say, "The milk is off". If by "the milk we had for breakfast", you the mean that actual milk that we ingested, then that is in our stomachs, and has been subjected to all sorts of enzymatic processes.
But I think the main determinant comes from a different context. That is, why are you saying it? If you want the milk thrown out, or some fresh milk bought, it's the first meaning, and you'd probably say "The milk we had is off". If one of you is feeling queasy, the meaning is the second, and you'd probably say, "The milk we had was off".