7
$\begingroup$

I have a dataset I'm working on that has some co-variate shift between the training set and the test set. I'm trying to build a predictive model to predict an outcome, using the training set. So far my best model is a random forest.

How can I deal with the shifted distributions in the training vs. test set? I've come across 2 possible solutions that I've been able to implement myself:

  1. Remove the shifted variables. This is sub-optimal, but helps prevent my model from over fitting the training set.
  2. Use a logistic regression to predict whether a given observation is from the test set (after balancing the classes), predict "test set probabilities" for the training set, and then boostrap sample the training set, using the probabilities for sampling. Then fit the final model on the new training set.

Both 1 and 2 are pretty easy to implement, but neither one satisfies me, as #1 omits variables that might be relevant, and #2 uses a logistic regression, when my final model is tree-based. Furthermore, #2 takes a few paragraphs of custom code, and I worry that my implementation may not be correct.

What are the standard methods for dealing with covariate shift? Are there any packages in R (or another language) that implement these methods?

/edit: It seems like "kernel mean matching" is another approach I could take. I've found lots of academic papers on the subject, but no one seems to have published any code. I'm going to try to implement this on my own, and will post the code as an answer to this question when I do.

$\endgroup$
7
$\begingroup$

Take a look at the twang package on CRAN. It casts the problem in terms of observational studies and propensity scoring, but the application is the same. Read the primary academic article for many more details and discussion: "Propensity Score Estimation with Boosted Regression for Evaluating Adolescent Substance Abuse Treatment".

It's really a tricky area though. If there is test/train overlap then twang (or other propensity score analysis) can help you out. If there are areas of no overlap then best of luck to you. Also, you don't need to resample, just weight your training data by Pr(Test)/Pr(Train). The derivation of that is covered in the article above.

$\endgroup$
3
$\begingroup$

I don't know of any packaged functions to perform correction for co-variate shift, but here is something that might work.

You could build a random-forest classifier between the training and test set (to estimate test-set probabilities). You can then learn a random-forest on the training set by sampling according to the estimated test-set probabilities. This will alleviate your concern number 2, by allowing a non-linear function learner. (By the way, you can also try the mboost package, which has several flavors of boosting implemented and is much faster than randomForest.)

Kernel mean matching may work but wouldn't necessarily be any better than any other non-linear classifier that outputs class probabilities -- and furthermore requires choices on the kernel and its hyper-parameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.