Suggestions to deal with a noisy testing dataset

Question

I'm actually training a model on a healthy training dataset to perform a regression task. To validate the model, I'm running it with different testing datasets. However, some testing datasets may suffer from a calibration default which creates a shift on the values of one ( or more ) feature(s). This leads sometimes to complete crashes of the model (I have tried LR, NN, RF and all of them are not robust against these crashes)

The problem is that I do not know in advance if the testing dataset is noisy or healthy. Moreover, I don't know in advance which feature may be affected (on some dataset it's the 3rd feature which is affected, on others it's the 1st one ..etc).

Do you have any suggestions to deal with this problem ? (ie. a strategy to compare the training and testing dataset to detect if there is a problem and correct it before running the model).

Answer 1

A good starting point would be to check the distributions of the features in the training and the testing dataset. If there are major differences in the distribution then the model performance is likely to collapse. For example: if you have a numerical feature like salary and its distribution varies a lot in the training and the testing dataset, then the model performance is likely to be bad on the testing set.

For numerical features, you can look at measures like range, mean, 25th, and 75th percentile.

The best way to deal with this problem would be to include such variations in the training set as well, then only you will be able to come up with a more robust model.