0
$\begingroup$

I am working on applying Random Forests to a multiclass classification problem, where I have a set of 11 predictor variables and a response that can take the values of "Yes", "No", and "Maybe". In my dataset, which has around 500 rows, I'd say about 95% of the data consists of "No" and "Maybe", there are very few "Yes" responses.

When I do the usual splitting of the data into the training and test sets, and then apply the Random Forests, no matter how many trees I have, my test error rate is always very good as it predicts the vast majority of the "No" and "Maybe" responses, but despite me trying many, many times, it NEVER predicts a single "Yes" response.

The "Yes" responses are obviously very important, they are not just "outliers", any idea why the random forests or boosting for that matter is unable to predict them?

$\endgroup$
  • $\begingroup$ The model can just predict not-Yes and be correct almost all of the time. It's hard to think of a penalty for a minimization scheme that would fix that and random sampling techniques will produce so many neo-samples with even further loss of "Yeses". $\endgroup$ – DWin May 4 '16 at 18:00
3
$\begingroup$

Your sample size is too low by a factor of at least 20 to be able to do split-sample validation.

As @user31264 said you need to concentrate on predicted probabilities, not forced choices.

Knowing that you need at least 96 observation just to estimate a single proportion (number of predictor variables = zero) it is highly unlikely that you have sufficient sample size in 500 to be able to fit a reliable multinomial logistic model with > 2 candidate predictors.

$\endgroup$
  • $\begingroup$ Hi. Thanks for your response. So, are you saying that if I know P(yes) and P(no), I can calculate P(maybe) = 1-P(yes)-P(no)? $\endgroup$ – Thomas Moore May 4 '16 at 23:18
  • $\begingroup$ Yes; a polytomous (multinomial) logistic model with 3 classes will give you 3 probabilities that sum to 1.0. See for example the R multinom function which I think is in the nnet package. $\endgroup$ – Frank Harrell May 5 '16 at 1:04
1
$\begingroup$

You need to predict the probabilities of "yes" and "no", and then calibrate them. Read about Platt's calibration and isotonic calibration.

$\endgroup$
  • $\begingroup$ Hi. Thanks for your response. So, are you saying that if I know P(yes) and P(no), I can calculate P(maybe) = 1-P(yes)-P(no)? $\endgroup$ – Thomas Moore May 4 '16 at 23:17
  • $\begingroup$ Yes, thats right. $\endgroup$ – user31264 May 5 '16 at 10:23
1
$\begingroup$

This is a common problem in rare events data (i.e., data where the event being modeled is relatively infrequent, < 5% of the data). Try random oversampling.

$\endgroup$
  • $\begingroup$ Thanks for your response. Can this be done in R? I'm using the randomForest library. $\endgroup$ – Thomas Moore May 4 '16 at 23:17
  • $\begingroup$ I'm not sure if there are native faculties in R for this, but it shouldn't be hard. Assign your event=1 values to one array and event=0 values to another. Then randomly sample with replacement your event=1 values into another array and merge the event=0 array with the result. You should then have a data set with event=1 and event=0 in equal proportions. Do your homework before proceeding on this, as there can be issues with cross-validating on improperly oversampled data (see marcoaltini.com/blog/…). $\endgroup$ – sw85 May 5 '16 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.