Classification model has bad results on new data

Question

I am working on a classification problem where I try to predict for each individual to which of two classes (0 or 1) they belong. The data set I have shows about 99% of individuals are in class 0, and 1% in class 1. The model I use is obtained by logistic regression, with a cut-off probability determined by ROC analysis. It performs decently on labeled test data (around 75% true positive and 25% false positive rate).

However, I also applied the obtained model on another unlabeled data set, and there many individuals get predicted to be in class 1. An inspection of the data shows that some risk factors such as age provide a higher risk in this set, which is the reason why the predicted probabilities are larger.

Is there anything I can do to cut down on the number of individuals predicted to be in class 1 (since it is unlikely so many are actually at such high risk)? I have tried to use a model based on features that are similar between the training data and the test set I'm trying to predict, but of course cutting down on the predictors we lose predictive power.

Any suggestions?

Answer 1

If the relationship between age and risk doesn't change with age, then the differences in distribution shouldn't matter, so I assume that it does change.

To illustrate, suppose you are fitting a linear relationship between age and height, using a sample of people uniformly distributed in 0-20 years of age. If you then apply this model to a new sample of only 15-20 years of age, you'll run into problems, because the relationship changes strongly with age - and your linear model doesn't account for that.

Conversely, if people did grow linearly 3 inches every year over their entire life, then it wouldn't make a difference whether the distribution differed between the training and test sample.

Now, you could either subsample the training set to match the age distribution you see in your test set, so the model learns the relationship that is actually present in the test set. Alternatively, you could attempt to learn the changing relationship, by including more flexibility in your model. Consider transforming age using splines. Or potentially consider interaction terms.

EDIT: here is an example. I couldn't find any data quickly, so I'll just simulate something. Below are 1000 pairs of "age" (say in years) and "height" (say in meters), plus a regression line in black - plus the true curved relationship in red.

Note that the regression line is determined on the entire training sample. If you were to apply this model to a test population aged only 60-80, then you'd overestimate their height systematically, since the black line is above the red one. The same for low ages. Conversely, in the age range 15-55, the regression actually overestimates height.

And note that this wouldn't happen so badly if the true relationship were in fact linear across the entire age range - then the red line would be straight, and close to the black one. Of course, the slopes could still differ, and the discrepancy resulting from slope differences would be largest for extreme values of age.

Two possible solutions, as I wrote above, would be to capture the true curved relationship via splines, or by matching the training and test sample on the relevant predictor.

set.seed(2)
nn <- 1e3
age.train <- sample(1:80,size=nn,replace=TRUE)
height.train <- age.train^(1/5)*0.6+0.6+rnorm(nn,0,0.2)
plot(age.train,height.train,pch=19,cex=0.6,col="grey")
abline(lm(height.train~age.train),lwd=2)
xx <- 1:80
lines(xx,xx^(1/5)*0.6+0.6,col="red",lwd=2)

Answer 2

When working with data that is not stable it can either be that the X is coming from different distributions or f(X) (being the function you are trying estimate) is non-stationary.

In the first case there are different solutions. First one use a classifier that captures allot of local structure. Thereby data that does not look like your test data will have lower influence and will not harm this as much. I recoment using a gradient boosting algorithm e.g. xgboost. Other than that you can weight the samples by how much they look like the test data in the training. This is often less critical for the model performance.

The second case where the label generating function changes is way more difficult. If you know a variable changes effect over time, then removing it seems like a solution. But do not it just because the variables changes overtime only if the dynamics changes. Otherwise i recoment using classifiers that does not rely on a few variables for their prediction. So for RandomForest lower the m_try/number_of_features and for the xgboost lower the colsample allot. This generally make the classifier more robust.

So what case do you think your in for this problem?

In general I have had the best luck with tree models when data distributions changes overtime. My intuition is that tree models perform 0-order interpolation making large outliers having lower effect than if for example using linear models.