Classification model has bad results on new data

I am working on a classification problem where I try to predict for each individual to which of two classes (0 or 1) they belong. The data set I have shows about 99% of individuals are in class 0, and 1% in class 1. The model I use is obtained by logistic regression, with a cut-off probability determined by ROC analysis. It performs decently on labeled test data (around 75% true positive and 25% false positive rate).

However, I also applied the obtained model on another unlabeled data set, and there many individuals get predicted to be in class 1. An inspection of the data shows that some risk factors such as age provide a higher risk in this set, which is the reason why the predicted probabilities are larger.

Is there anything I can do to cut down on the number of individuals predicted to be in class 1 (since it is unlikely so many are actually at such high risk)? I have tried to use a model based on features that are similar between the training data and the test set I'm trying to predict, but of course cutting down on the predictors we lose predictive power.

Any suggestions?

• Does the distribution of age differ between the datasets, or the impact of age on risk? (Or both?) – Stephan Kolassa May 30 '17 at 8:05
• Any classification and prediction operates under the assumption that the relationships between predictors and outcomes are similar in the training and the test set. ("Relationships" can of course include modelable dynamics like trend or seasonality.) So if you know beforehand about the differences between the training and test set (see my previous comment), you could subsample the training set and build a model based on instances that are similar to the test set. – Stephan Kolassa May 30 '17 at 8:09
• The distribution of age (mostly the location) differs between the large labeled data (80000 rows) and the sizeable but not as large (2000 rows) unlabeled data. Also the distribution of the other predictors differs, which is most likely why the results are so different. – Sanderr May 30 '17 at 8:10

If the relationship between age and risk doesn't change with age, then the differences in distribution shouldn't matter, so I assume that it does change.

To illustrate, suppose you are fitting a linear relationship between age and height, using a sample of people uniformly distributed in 0-20 years of age. If you then apply this model to a new sample of only 15-20 years of age, you'll run into problems, because the relationship changes strongly with age - and your linear model doesn't account for that.

Conversely, if people did grow linearly 3 inches every year over their entire life, then it wouldn't make a difference whether the distribution differed between the training and test sample.

Now, you could either subsample the training set to match the age distribution you see in your test set, so the model learns the relationship that is actually present in the test set. Alternatively, you could attempt to learn the changing relationship, by including more flexibility in your model. Consider transforming age using . Or potentially consider terms.

EDIT: here is an example. I couldn't find any data quickly, so I'll just simulate something. Below are 1000 pairs of "age" (say in years) and "height" (say in meters), plus a regression line in black - plus the true curved relationship in red.

Note that the regression line is determined on the entire training sample. If you were to apply this model to a test population aged only 60-80, then you'd overestimate their height systematically, since the black line is above the red one. The same for low ages. Conversely, in the age range 15-55, the regression actually overestimates height.

And note that this wouldn't happen so badly if the true relationship were in fact linear across the entire age range - then the red line would be straight, and close to the black one. Of course, the slopes could still differ, and the discrepancy resulting from slope differences would be largest for extreme values of age.

Two possible solutions, as I wrote above, would be to capture the true curved relationship via , or by matching the training and test sample on the relevant predictor.

set.seed(2)
nn <- 1e3
age.train <- sample(1:80,size=nn,replace=TRUE)
height.train <- age.train^(1/5)*0.6+0.6+rnorm(nn,0,0.2)
plot(age.train,height.train,pch=19,cex=0.6,col="grey")
abline(lm(height.train~age.train),lwd=2)
xx <- 1:80
lines(xx,xx^(1/5)*0.6+0.6,col="red",lwd=2)


• If increased age induces increased risk (like for example a survival model), then certainly if the new data contains many people with a high age, the overall predicted risk for this data will be high. Or is this not what you mean with "the differences in distribution shouldn't matter"? – Sanderr May 30 '17 at 8:26
• Yes, if the risk is higher because people are older, then predicted risks will be higher - but this would be correct, and you wouldn't be asking here. What your problem seems to be is that the relationship you learn in your training set differs systematically from the relationship present in your test set. Specifically, your model based on the training set overestimates the risk for high levels of age. Which is why I'd recommend either matching/subsampling, or splines. Let me see whether I can find some data and edit my answer. – Stephan Kolassa May 30 '17 at 8:42

When working with data that is not stable it can either be that the X is coming from different distributions or f(X) (being the function you are trying estimate) is non-stationary.

In the first case there are different solutions. First one use a classifier that captures allot of local structure. Thereby data that does not look like your test data will have lower influence and will not harm this as much. I recoment using a gradient boosting algorithm e.g. xgboost. Other than that you can weight the samples by how much they look like the test data in the training. This is often less critical for the model performance.

The second case where the label generating function changes is way more difficult. If you know a variable changes effect over time, then removing it seems like a solution. But do not it just because the variables changes overtime only if the dynamics changes. Otherwise i recoment using classifiers that does not rely on a few variables for their prediction. So for RandomForest lower the m_try/number_of_features and for the xgboost lower the colsample allot. This generally make the classifier more robust.

So what case do you think your in for this problem?

In general I have had the best luck with tree models when data distributions changes overtime. My intuition is that tree models perform 0-order interpolation making large outliers having lower effect than if for example using linear models.