I'm training a classification model with Random Forest to discriminate between 6 categories. My transactional data has approximately 60k+ observations and 35 variables. Here's an example of how it approximately looks like.

 _________________________________________________
|user_id|acquisition_date|x_var_1|x_var_2| y_vay  |
|-------|----------------|-------|-------|--------|
|111    | 2013-04-01     | 12    | US    | group1 |
|222    | 2013-04-12     | 6     | PNG   | group1 |
|333    | 2013-05-05     | 30    | DE    | group2 |
|444    | 2013-05-10     | 78    | US    | group3 |
|555    | 2013-06-15     | 15    | BR    | group1 |
|666    | 2013-06-15     | 237   | FR    | group6 |

Once the the model is created, I'd like to score observations from the last few week. As there have been changes to the system, the more recent observations will resemble more closely the environment of the current observations that I'd like to predict. Hence, I want to create a weight variable so that the Random Forest would put more importance on the recent observations.

Does anyone know if the randomForest package in R able to handle weights per observation?

Also, can you please suggest what is a good method for creating the weight variable? For example, as my data is from 2013, I was thinking that I can take the month number from the date as weight. Does anyone see a problem with this method?

Many thanks in advance!

  • You might consider posting a question on stack overflow. They will help you more with implementation issues. The focus of this site is more theory-based. – Alex Williams Jan 23 '14 at 13:59
  • I am probably not clear enough in my writing, but my questions are not regarding implementation issue. For example, in the part where I'm asking about creating the weight variable, I don't mean to ask what command in R can help me do that. I was simply wondering if by that I would be violating any of the assumptions of the random forest. – Nikolay Nenov Jan 23 '14 at 15:57
  • 1
    The answer is no, I believe. You can assign weights to different groups as I explained in my answer below. I understand this is not what you are interested in, but it is a similar idea. You could try using duplicate observations as I suggest. – Alex Williams Jan 23 '14 at 18:17
up vote 5 down vote accepted

The ranger package in R (pdf), which is relatively new, will do this. The ranger implementation of random forests has a case.weights argument that takes a vector with individual case / observation weights.

  • Neat! Seams like the solution I was looking for. Do you have a link to the details of how the probability is calculated case.weights? – Nikolay Nenov Jun 29 '17 at 14:17
  • 1
    I'm not 100% sure how they calculate the probabilities - but I think, if you want a start, have a look at this paper: Malley, J. D., Kruppa, J., Dasgupta, A., Malley, K. G., & Ziegler, A. (2012). Probability machines: consistent probability estimation using nonparametric learning machines. Methods Inf Med 51:74-81. dx.doi.org/10.3414/ME00-01-0052 – GrantRWHumphries Jun 30 '17 at 14:39

You could resample the data to over represent the more recent data points. Rf involves a sampel-with-replacment step anyways and "roughly balanced bagging" for unbalanced classes uses sampling to overrepresent the minority class and produces results as good or better then class weighted random forest in my experience.

You could resample at the level of constructing your training matrix (reference) instead of during bagging to keep implementation easy though I would suggest doing many repeats in that case.

Internally some implementations of random forest including scikit-learn actually use sample weights to keep track of how many times each sample is in bag and it should be equivalent to oversampling at the bagging level and close to oversampling at the training level in cross validation.

You should look into the "classwt" parameter. This doesn't seem to be what you are directly interested in, but it might give you a sense of what you want to do.

See here: Stack Exchange question #1

And here: Stack Exchange question #2

Article on weighted random forests: PDF

The basic idea is to weight classes such that rarely observed groups/classifications are more likely to be selected in your bootstrap samples. This is helpful for imbalanced data (when the prior probabilities of different classes are widely different).

It seems to me that you want to do something similar, but for recent events (not for certain groups/classifications). A simple way to do this would be to create duplicate observations (i.e. put in repeated, identical rows) for more recent observations. However, this could potentially be inefficient. I do not know of a way to directly weight each observation in R, but I could be unaware of it.

You could try looking around for alternative implementations, e.g. in C -- at worst, these could be customized with a bit of coding.

  • 1
    Thanks a lot for the links, Alex. The paper gives good examples of cases in which you'd want to weight your classifiers. I'm afraid this is not working for me, though, as one cannot use the "classwt" parameter for anything other than weighting the classifiers - i.e. you need one weight per class, otherwise the randomForest would return an error. – Nikolay Nenov Jan 23 '14 at 15:52
  • 1
    Yes I don't think you can use "classwt" directly. You want some parameter like "observationweights" but I don't think it exists. – Alex Williams Jan 23 '14 at 18:18

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.