I have a dataset I'm working on that has some co-variate shift between the training set and the test set. I'm trying to build a predictive model to predict an outcome, using the training set. So far my best model is a random forest.
How can I deal with the shifted distributions in the training vs. test set? I've come across 2 possible solutions that I've been able to implement myself:
- Remove the shifted variables. This is sub-optimal, but helps prevent my model from over fitting the training set.
- Use a logistic regression to predict whether a given observation is from the test set (after balancing the classes), predict "test set probabilities" for the training set, and then boostrap sample the training set, using the probabilities for sampling. Then fit the final model on the new training set.
Both 1 and 2 are pretty easy to implement, but neither one satisfies me, as #1 omits variables that might be relevant, and #2 uses a logistic regression, when my final model is tree-based. Furthermore, #2 takes a few paragraphs of custom code, and I worry that my implementation may not be correct.
What are the standard methods for dealing with covariate shift? Are there any packages in R (or another language) that implement these methods?
/edit: It seems like "kernel mean matching" is another approach I could take. I've found lots of academic papers on the subject, but no one seems to have published any code. I'm going to try to implement this on my own, and will post the code as an answer to this question when I do.