I got good results on a radiation exposure prediction problem using SVM and DT where the ultimate goal is to predict the radiation dose an individual was exposed to using data about individual related to their health.

Overall, the feedback was positive but one comment was interesting to me: How does one ensure Machine Learning doesn't come to correct classifications via the wrong ways? How did you exclude this?

I took all the necessary precautions to make sure things like class imbalance, overfitting, etc. weren't an issue, so I'm certainly under the impression this is more related to the fundamentals of the field rather than an ML methodology question.

How would one begin to approach this question from an ML perspective where I feel the answer basically boils down to the fact that, after doing all necessary procedures that can be done to optimize model performance (regularization, cross validation, etc.), there is no such thing as "coming to the right classifications the wrong way"?

I feel like this is actually a question that needs a good answer for any technical communication to audiences perhaps not as conversant with ML in general (or who perhaps are but want to hear our thoughts nonetheless).

  • 4
    $\begingroup$ (+1) Reaching a correct conclusion for the wrong reasons is actually a big problem, and an active area of research. For instance, you can trick a DNN into misidentifying images by adding certain noise patterns (see: stats.stackexchange.com/questions/344498/…) There are related problems pertaining to biases in data collection. But also this problem isn't unique advanced machine learning methods; model mis-specification can reverse the sign in a regression coefficient! $\endgroup$
    – Sycorax
    Jul 26, 2021 at 14:00
  • 1
    $\begingroup$ Unbalanced classes are almost certainly not a problem, so you don't need to take "necessary precautions" (beyond using appropriate evaluation measures, in particular not accuracy): Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? $\endgroup$ Jul 26, 2021 at 14:08
  • 2
    $\begingroup$ Sycorax's mention of adversarial examples and various ways of tricking ML is a good start. But I'm still a bit unclear about what you mean by "wrong" ways to arrive at a classification/prediction. Presumably it's stuff like "if the background is white, then classify the dog image as a husky". But I'll play devil's advocate here: what is problematic or "wrong" about using context in your classification? If I see a dog in snow, it is a priori more likely to be a husky than a pekinese. So: what would a "wrong" way be? $\endgroup$ Jul 26, 2021 at 14:12
  • 1
    $\begingroup$ @Prospero Both F1 and MCC are threshold-based metrics that have many of the same issues as accuracy. $\endgroup$
    – Dave
    Jul 26, 2021 at 14:43
  • 2
    $\begingroup$ On evaluating predictions for "unbalanced" data, see the link in my previous comment, also Is accuracy an improper scoring rule in a binary classification setting? Classification probability threshold. Use probabilistic classifications, and evaluate these using proper scoring rules. $\endgroup$ Jul 26, 2021 at 19:22

3 Answers 3


One problem in ML is when it uses predictors we do not want it to use, like gender or ethnicity. And even if these are not fed into the model, we may still have predictors that do correlate with these factors, like ZIP codes correlating with ethnicity, or colleges correlating with gender.

Assuming that, say, gender does correlate with the outcome we are modeling, then even if we do not feed in genders, but only the college attended, and some colleges are traditionally gender-imbalanced, then we will overall get different classifications or predictions for men than for women.

This particular case can be found out by slicing you dataset by gender and checking the outcome predictions, while ignoring all other pieces of information.

Unfortunately, this is not simple to do, because the model is not using gender (which we didn't feed in). It's using the college attended, which in turn is correlated with the outcome. Does it even make sense to only slice the dataset in a single dimension like gender, while ignoring possible mediators or confounders, like the college? Is the problem that students from college A perform worse, and that men predominantly attend college A, or is the problem that men perform worse, and that they predominantly attend college A? And which predictor represents a "wrong" way for the model to come to predictions?

And then, of course, all this is mixed up with the question of whether the original problem is that the training data already exhibits the results of bias. Maybe male students from college A historically performed worse because there was always a hiring bias against students from college A. Or, conversely, a bias against men. There is no easy solution to this, because it is rarely possible to tease out the "real" effect of bias in the training sample from any true underlying differences.

Bottom line: there is no simple way to find out whether your model arrived at the "correct" predictions ("men perform worse") through "correct" ways (men indeed perform worse) or through "wrong" ways (men predominantly attend college A, and students from college A perform worse). In particular, there is no way you could test programmatically. Your best bet is likely to subject your model to various stress tests, and have a plan on how to react if you go into production and someone detects a flaw you didn't think of.

  • 3
    $\begingroup$ (+1) I've often seen this important fact summarized as "machine learning is really good at inferring latent variables, so it can make (problematic) inferences about latent categories even when it does not have access to that information directly." $\endgroup$
    – Sycorax
    Jul 26, 2021 at 14:38
  • $\begingroup$ +1 as well. This explains the dilemma well. I think none of us are satisfied with the answer, but this is the best answer we have at this juncture. $\endgroup$
    – Prospero
    Jul 27, 2021 at 15:41

+1: extremely deep question!

I will repeat the advice I got from my advisor, without necessarily understanding it! This was years ago so I am probably misrepresenting it.

The problem was classification at the presence of some obvious noise (negative signal power levels). I asked: "How should I filter the noise out?". He answered "You don't. Firstly, the ML model is going to do that for you. Secondly, the noise profile is valuable training data for the model."

I am not happy with that but do believe he's extremely competent at what he does.


This is the purpose of the Validation Set. Split your dataset in 3 : Train, Test and Validation. Never touch your Validation set again until the last phase. Create your model using Train and Test, train your encoders, make your preprocesses, create your variables etc. Then create your model and tune it using Train and Test.

When you're satisfied and have the model you want, then apply your encoders, preprocesses on your Validation set, which acts as totally new and unknown Data from the model, the same type as you'd have on a real case. Apply your model on those brand new data (that you didn't use as a reference to train your model, as you did with Test) to have an overlook on how your model will perform in real time with new data. This can be shown easily to the audience : Ask them to give you the latest examples they have, remove the final answer from the data, and run your model on it. You'll have precise results on how your model works on fresh data. If they're still doubting, ask for a test phase in which you run your algorithm day to day and in real time, without replacing their current system for the moment, so you can compare results at the end of the test phase.

Another thing to check is being sure you don't use variables that you shouldn't have in real case, or that you shouldn't know without knowing the target. That's a classic case of having a good classification from a wrong way.

  • $\begingroup$ Just to note that the process you call "Train-Test-Validation" many others call "Train-(Cross)Validation-Test" and the end of your second paragraph seems to be heading in that direction. $\endgroup$
    – Henry
    Jul 26, 2021 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.