Weighting more recent data in Random Forest model

Question

I'm training a classification model with Random Forest to discriminate between 6 categories. My transactional data has approximately 60k+ observations and 35 variables. Here's an example of how it approximately looks like.

 _________________________________________________
|user_id|acquisition_date|x_var_1|x_var_2| y_vay  |
|-------|----------------|-------|-------|--------|
|111    | 2013-04-01     | 12    | US    | group1 |
|222    | 2013-04-12     | 6     | PNG   | group1 |
|333    | 2013-05-05     | 30    | DE    | group2 |
|444    | 2013-05-10     | 78    | US    | group3 |
|555    | 2013-06-15     | 15    | BR    | group1 |
|666    | 2013-06-15     | 237   | FR    | group6 |

Once the the model is created, I'd like to score observations from the last few week. As there have been changes to the system, the more recent observations will resemble more closely the environment of the current observations that I'd like to predict. Hence, I want to create a weight variable so that the Random Forest would put more importance on the recent observations.

Does anyone know if the randomForest package in R able to handle weights per observation?

Also, can you please suggest what is a good method for creating the weight variable? For example, as my data is from 2013, I was thinking that I can take the month number from the date as weight. Does anyone see a problem with this method?

Many thanks in advance!

Answer 1

The ranger package in R (pdf), which is relatively new, will do this. The ranger implementation of random forests has a case.weights argument that takes a vector with individual case / observation weights.

Answer 2

You could resample the data to over represent the more recent data points. Rf involves a sampel-with-replacment step anyways and "roughly balanced bagging" for unbalanced classes uses sampling to overrepresent the minority class and produces results as good or better then class weighted random forest in my experience.

You could resample at the level of constructing your training matrix (reference) instead of during bagging to keep implementation easy though I would suggest doing many repeats in that case.

Internally some implementations of random forest including scikit-learn actually use sample weights to keep track of how many times each sample is in bag and it should be equivalent to oversampling at the bagging level and close to oversampling at the training level in cross validation.

Answer 3

You should look into the "classwt" parameter. This doesn't seem to be what you are directly interested in, but it might give you a sense of what you want to do.

See here: Stack Exchange question #1

And here: Stack Exchange question #2

Article on weighted random forests: PDF

The basic idea is to weight classes such that rarely observed groups/classifications are more likely to be selected in your bootstrap samples. This is helpful for imbalanced data (when the prior probabilities of different classes are widely different).

It seems to me that you want to do something similar, but for recent events (not for certain groups/classifications). A simple way to do this would be to create duplicate observations (i.e. put in repeated, identical rows) for more recent observations. However, this could potentially be inefficient. I do not know of a way to directly weight each observation in R, but I could be unaware of it.

You could try looking around for alternative implementations, e.g. in C -- at worst, these could be customized with a bit of coding.