Co-variate shift between train and test data-set What are some of the modern techniques used to detect and mitigate the covariate shift between train and test datasets? 
One method I recently read involves training a classifier using Kullback-Leibler divergence loss function, this loss function is recommended since it measures the entropy or randomness. After training the classifier, test the model on the test data and calculate the Matthews correlation coefficient. As far as I know, Matthews correlation coefficient is used to estimate the quality of the classifier. Mathematically it just another measure of true/false positives and true/false negatives. How can we use this to detect the covariate shift? 
If we suspect there is a co-variate shift, what are some of the recommended methods to mitigate this effect? Would a random mix of train and test set help to solve the problem? Is that a common approach? 
 A: The problem of covariate shift or concept drift is addressed in the fields of medical statistics, domain adaption, transfer learning, and online learning (learning from streams). Therefore, the literature is exhaustive. Reviews from the different domains are below:

*

*Jiang, Jing. "A Literature Survey on Domain Adaptation of Statistical Classifiers." (2008)

*Pan, Sinno Jialin, and Qiang Yang. "A Survey on Transfer Learning." (2010)

*Ditzler, Gregory, et al. "Learning in nonstationary environments: A survey." (2015)

*Janssen, K. J. M., et al. "Updating methods improved the performance of a clinical prediction model in new patients." (2008)

*Sugiyama, Masashi, Neil D. Lawrence, and Anton Schwaighofer. Dataset shift in machine learning. The MIT Press, 2017.

Usually, some sort of re-weighting is required to compensate for covariate shift. The objective is to transform the shifted distribution in the test data to match the distribution that was observed during training. For clinical risk models, which most of the time are linear, this is called recalibration and involves re-weighting the intercept and coefficients by a factor (see review by Janssen et al.). In the machine learning domain, a popular approach is to re-weight instances and train a model on the updated data. There are a myriad of approaches to estimate these weights, importance sampling and kernel mean matching are two classical examples.
