Unless there was a particular context to warrant this variation, I don't think it would work.
Idioms are set phrases, and only a few have some accepted variations.
For example, consider changing just "tea" to "coffee" in that idiom. "It's not really my cup of coffee." That doesn't really work for me.
Interested in Language