How does one ensure Machine Learning doesn't come to correct classifications via the wrong ways?

Question

I got good results on a radiation exposure prediction problem using SVM and DT where the ultimate goal is to predict the radiation dose an individual was exposed to using data about individual related to their health.

Overall, the feedback was positive but one comment was interesting to me: How does one ensure Machine Learning doesn't come to correct classifications via the wrong ways? How did you exclude this?

I took all the necessary precautions to make sure things like class imbalance, overfitting, etc. weren't an issue, so I'm certainly under the impression this is more related to the fundamentals of the field rather than an ML methodology question.

How would one begin to approach this question from an ML perspective where I feel the answer basically boils down to the fact that, after doing all necessary procedures that can be done to optimize model performance (regularization, cross validation, etc.), there is no such thing as "coming to the right classifications the wrong way"?

I feel like this is actually a question that needs a good answer for any technical communication to audiences perhaps not as conversant with ML in general (or who perhaps are but want to hear our thoughts nonetheless).

Answer 1

One problem in ML is when it uses predictors we do not want it to use, like gender or ethnicity. And even if these are not fed into the model, we may still have predictors that do correlate with these factors, like ZIP codes correlating with ethnicity, or colleges correlating with gender.

Assuming that, say, gender does correlate with the outcome we are modeling, then even if we do not feed in genders, but only the college attended, and some colleges are traditionally gender-imbalanced, then we will overall get different classifications or predictions for men than for women.

This particular case can be found out by slicing you dataset by gender and checking the outcome predictions, while ignoring all other pieces of information.

Unfortunately, this is not simple to do, because the model is not using gender (which we didn't feed in). It's using the college attended, which in turn is correlated with the outcome. Does it even make sense to only slice the dataset in a single dimension like gender, while ignoring possible mediators or confounders, like the college? Is the problem that students from college A perform worse, and that men predominantly attend college A, or is the problem that men perform worse, and that they predominantly attend college A? And which predictor represents a "wrong" way for the model to come to predictions?

And then, of course, all this is mixed up with the question of whether the original problem is that the training data already exhibits the results of bias. Maybe male students from college A historically performed worse because there was always a hiring bias against students from college A. Or, conversely, a bias against men. There is no easy solution to this, because it is rarely possible to tease out the "real" effect of bias in the training sample from any true underlying differences.

Bottom line: there is no simple way to find out whether your model arrived at the "correct" predictions ("men perform worse") through "correct" ways (men indeed perform worse) or through "wrong" ways (men predominantly attend college A, and students from college A perform worse). In particular, there is no way you could test programmatically. Your best bet is likely to subject your model to various stress tests, and have a plan on how to react if you go into production and someone detects a flaw you didn't think of.

Answer 2

+1: extremely deep question!

I will repeat the advice I got from my advisor, without necessarily understanding it! This was years ago so I am probably misrepresenting it.

The problem was classification at the presence of some obvious noise (negative signal power levels). I asked: "How should I filter the noise out?". He answered "You don't. Firstly, the ML model is going to do that for you. Secondly, the noise profile is valuable training data for the model."

I am not happy with that but do believe he's extremely competent at what he does.

Answer 3

This is the purpose of the Validation Set. Split your dataset in 3 : Train, Test and Validation. Never touch your Validation set again until the last phase. Create your model using Train and Test, train your encoders, make your preprocesses, create your variables etc. Then create your model and tune it using Train and Test.

When you're satisfied and have the model you want, then apply your encoders, preprocesses on your Validation set, which acts as totally new and unknown Data from the model, the same type as you'd have on a real case. Apply your model on those brand new data (that you didn't use as a reference to train your model, as you did with Test) to have an overlook on how your model will perform in real time with new data. This can be shown easily to the audience : Ask them to give you the latest examples they have, remove the final answer from the data, and run your model on it. You'll have precise results on how your model works on fresh data. If they're still doubting, ask for a test phase in which you run your algorithm day to day and in real time, without replacing their current system for the moment, so you can compare results at the end of the test phase.

Another thing to check is being sure you don't use variables that you shouldn't have in real case, or that you shouldn't know without knowing the target. That's a classic case of having a good classification from a wrong way.