About the main question, it depends which milk is actually being spoken of. If by "the milk we had for breakfast", you mean the container of milk from which we poured our own," you can say, "The milk is off". If by "the milk we had for breakfast", you the mean that actual milk that we ingested, then that is in our stomachs, and has been subjected to all sorts of enzymatic processes.
But I think the main determinant comes from a different context. That is, why are you saying it? If you want the milk thrown out, or some fresh milk bought, it's the first meaning, and you'd probably say "The milk we had is off". If one of you is feeling queasy, the meaning is the second, and you'd probably say, "The milk we had was off".