Test data distribution different than training data distribution

Question

What could be some issues if the distribution of the test data is significantly different than the distribution of the training data and why is that?

## Edit 1 ##

The training and testing sets came from different processes. I was thinking if I use a set of data, that is say normally distributed, to train and build a model (e.g. logistic regression) and then use that model to make predictions on a new set of data, say it is right skewed. What could be the issues here and how would it impact the predictions?

Answer 1

I'm going to make an assumption here: When you say "the test data is significantly different from the training data" I am taking you to mean that for some reason you ran a goodness of fit test (e.g., Kolmogorov Two-sample test) and rejected the null. If that isn't what you mean, what do you mean? If that is what you mean, what test did you run (and why)?

So, given the assumption, the first question I would ask is, "Could this be a case of rejecting a true null?" To answer that question, I need to know how the training and test data came to be. If the training data are a random sample of the full data, then very clearly this IS a case of rejecting a true null.

If they came into existence via different processes, then you've established that the training data and test data are different. To me, that means that your classification (prediction) model is probably biased. Think about it in terms of a simple linear regression model. If you got your training data on the relationship to predict H.S. GPA fom IQ at a Mensa chapter, would you apply the model to the kids at Central High School? CHS students almost surely have a broader range of IQ scores than the Mensa chapter. The Mensa chapter almost certainly has a broader range of ages present than does CHS. You have to do serious extrapolation to apply the model estimates to CHS.

You have to take it on faith that the extrapolation is valid, that is, that the model you fit continues to hold in the new circumstances. So, my first issue is model bias (lack of fit bias of a sort) and that by itself would make me avoid applying the model to the test data.

Answer 2

It is an research topic in domain adaptation. "the distribution of the test data is significantly different than the distribution of the training data" may have following meanings:

• p(x,y) are different
• p(x) are different, p(y|x) are the same
• p(x) are the same, p(y|x) are different.

You can find many papers with the key words "domain adaptation". It seems that there are no perfect solutions.