3
$\begingroup$

I am applying Random Forest to a matrix of 388 samples by 14 features.

The features are:

  • nominal (5 categories) (1 feature)
  • nominal (2 categories) (13 features)

The target variable is nominal (6 categories).

Computing the out-of-bag score I get a score of 0.4974, which means, if I understood well, that my classifier misclassifies half of the samples.

I am using 1000 trees, which are expanded until all leaves are composed by only 1 sample.

I am using the Random Forest implementation in Scikit-learn.

What am I doing wrong? I was thinking that maybe the number of samples is too large with respect to the number of features, but I am unsure on that.

$\endgroup$
9
  • 3
    $\begingroup$ It's quite likely that you are doing nothing wrong and that the data is sufficiently diffuse so as not to permit a more accurate analysis. Describe your target variable. What are the features like? Even before you run your RFs, what does a simple, pairwise exploratory analysis tell you about the dependence between features and target? $\endgroup$
    – user78229
    Commented Nov 13, 2015 at 10:14
  • $\begingroup$ I've updated the question. What do you mean with pairwise exploratory analysis? Thanks $\endgroup$
    – gc5
    Commented Nov 13, 2015 at 10:28
  • 1
    $\begingroup$ How many different categories are there in the target variable ? 50% can be a good performance with a high number of classes $\endgroup$
    – RUser4512
    Commented Nov 13, 2015 at 10:37
  • $\begingroup$ Thanks. "Boolean" isn't too informative a statement about the features. If your target is categorical, is it multinomial, nominal or ordinal in scaling? "Pairwise" association refers to developing metrics appropriate for the structure of the dependence for all of the possible combinations between your target and each of the features. This could be a Spearman correlation if the target is ordinal or a discrete metric such as Somer's D if it's nominal. Basically, the maximum value in magnitude across the set of possible combinations should give you a rough upper bound in terms for your RF $\endgroup$
    – user78229
    Commented Nov 13, 2015 at 10:38
  • $\begingroup$ RUser4512: in the target variable there are 6 categories. DJohnson: sorry, what could be used instead of boolean? The target is composed by 6 categories, so I think it could be considered nominal variable (can you explain me multinomial?). Ok for the pairwise association, I will go for Somer's D. $\endgroup$
    – gc5
    Commented Nov 13, 2015 at 10:52

1 Answer 1

2
$\begingroup$

In worst case scenario, your features are completely unrelated to your target or related to your target in such a complicated way, that even RF cannot learn any reproducible pattern. A RF model will as default(no class weight, no stratification) assume same target distribution as of training set. If e.g. the most prevalent class make up 80% of the training set, the RF model can still use this information alone to roughly predict any new sample as member of this class, and achieve only a 20% class err.rate. In your case, if your training data is balanced such that each class is represented by 100%/6=16.7%, the expected worst performance is cross-validated 83.3% err.rate.

"What am I doing wrong?" - probably nothing, just too poor variables to predict your target any better. Try some 'feature engineering' or get some new variables. Maybe you realize 50% err.rate for your problem is not that bad at all. If you could predict with 50% err.rate the winner of the next 50 Tennis grand-slam tournaments, you probably could earn a fortune on sport betting.

"I was thinking that maybe the number of samples is too large with respect to the number of features" -That is never a problem. Feel free to discard a random fraction of your samples. It won't make your model better though.

I can warmly recommend the tutorial competition at kaggle: Titanic, which can teach you how to assess your RF model performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.