Although I'm not a teacher, I would say that "subtleness" is not in very common usage at all. A comparison of Google hits for example, shows around 81,000 for "subtleness" and around 3.65 million for "subtlety". The OED's entry for "subtleness" consists of single word in defintion - subtlety. This strengthens my conviction that they are effectively exact synonyms, and that the optimal choice would "subtlety" over "subtleness".