3
$\begingroup$

What are some of the modern techniques used to detect and mitigate the covariate shift between train and test datasets?

One method I recently read involves training a classifier using Kullback-Leibler divergence loss function, this loss function is recommended since it measures the entropy or randomness. After training the classifier, test the model on the test data and calculate the Matthews correlation coefficient. As far as I know, Matthews correlation coefficient is used to estimate the quality of the classifier. Mathematically it just another measure of true/false positives and true/false negatives. How can we use this to detect the covariate shift?

If we suspect there is a co-variate shift, what are some of the recommended methods to mitigate this effect? Would a random mix of train and test set help to solve the problem? Is that a common approach?

$\endgroup$

1 Answer 1

2
$\begingroup$

The problem of covariate shift or concept drift is addressed in the fields of medical statistics, domain adaption, transfer learning, and online learning (learning from streams). Therefore, the literature is exhaustive. Reviews from the different domains are below:

Usually, some sort of re-weighting is required to compensate for covariate shift. The objective is to transform the shifted distribution in the test data to match the distribution that was observed during training. For clinical risk models, which most of the time are linear, this is called recalibration and involves re-weighting the intercept and coefficients by a factor (see review by Janssen et al.). In the machine learning domain, a popular approach is to re-weight instances and train a model on the updated data. There are a myriad of approaches to estimate these weights, importance sampling and kernel mean matching are two classical examples.

$\endgroup$
1
  • $\begingroup$ An issue is how did the training and test sets arise? In most cases creation of such splits are not advised and a resampling validation method recommended. $\endgroup$ Aug 16, 2022 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.