You could, but it is not usually put that way. In any case, "I'm off tomorrow" (at least, in Canada) usually means "I will be gone tomorrow" (on a trip, for example). If you mean that you are not coming in to work (again, in Canada) you would usually say " I won't be in tomorrow"