22
$\begingroup$

I'm experimenting with random forests with scikit-learn and I'm getting great results of my training set, but relatively poor results on my test set...

Here is the problem (inspired from poker) which I'm trying to solve: Given player A's hole cards, player B's hole cards and a flop (3 cards), which player has the best hand? Mathematically, this is 14 inputs (7 cards -- one rank and one suit for each) and one output (0 or 1).

Here are some of my results so far:

Training set size: 600k, test set size: 120k, number of trees: 25
Success rate in training set: 99.975%
Success rate in testing set: 90.05%

Training set size: 400k, test set size: 80k, number of trees: 100
Success rate in training set: 100%
Success rate in testing set: 89.7%

Training set size: 600k, test set size: 120k, number of trees: 5
Success rate in training set: 98.685%
Success rate in testing set: 85.69%

Here is the relevant code used:

from sklearn.ensemble import RandomForestClassifier
Forest = RandomForestClassifier(n_estimators = 25) #n_estimator varies
Forest = Forest.fit(inputs[:trainingSetSize],outputs[:trainingSetSize])
trainingOutputs = Forest.predict(inputs[:trainingSetSize])
testOutputs = Forest.predict(inputs[trainingSetSize:])

It appears that regardless of the number of trees used, performance on training set is much better than on test set, despite a relatively large training set and a reasonably small number of features...

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ I'm not seeing about a dozen of the 'dials' for random forests here. Cross Validation? Bayesian Priors? Nature of the resampling? Training set for each tree? What percent of the subset to each tree? ... there are lots more that could be enumerated, but my point is that you have other inputs to consider. $\endgroup$
    – EngrStudent
    Aug 5, 2013 at 0:18
  • 1
    $\begingroup$ Could you explain the problem for those who don't know poker.. is there an easy calculation for the poker score? then its easier to understand whether there is something fundamentally wrong with using RF... I don't know poker, but I suspect RF probably is the wrong approach - namely the first step IN RF is to use only a fraction of the inputs, whereas it would appear to me that there is no way of building a good classifier using just a subset of the inputs - all the inputs are required. $\endgroup$
    – seanv507
    Aug 5, 2013 at 14:38

2 Answers 2

Reset to default
49
$\begingroup$

This is a common rookie error when using RF models (I'll put my hand up as a previous perpetrator). The forest that you build using the training set will in many cases fit the training data almost perfectly (as you are finding) when considered in totality. However, as the algorithm builds the forest it remembers the out-of-bag (OOB) prediction error, which is its best guess of the generalization error.

If you send the training data back into the predict method (as you are doing) you get this almost perfect prediction (which is wildly optimistic) instead of the correct OOB error. Don't do this. Instead, the trained Forest object should have remembered within it the OOB error. I am unfamiliar with the scikit-learn implementation but looking at the documentation here it looks like you need to specify oob_score=True when calling the fit method, and then the generalization error will be stored as oob_score_ in the returned object. In the R package "randomForest", calling the predict method with no arguments on the returned object will return the OOB prediction on the training set. That lets you define the error using some other measure. Sending the training set back into the predict method will give you a different result, as that will use all the trees. I don't know if the scikit-learn implementation will do this or not.

It is a mistake to send the training data back into the predict method in order to test the accuracy. It's a very common mistake though, so don't worry.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Thanks! However, I still have a concern: With 400k training examples and 50 trees, I got 89.6% correct, whereas with as much data and twice as many trees, I got 89.7% correct...Does this suggest that RF is not a good method for this? I used a MLP neural network in the past and reached a ~98.5% accuracy on the testing set... $\endgroup$
    – Uwat
    Aug 5, 2013 at 2:30
  • 5
    $\begingroup$ That's possible, although it looks like you are not using nearly enough trees. Typically you need thousands. Note that number of trees is not a parameter to be tuned in the RF algorithm, more is always better, but once you have 'enough' (to be determined empirically) the OOB error doesn't improve with more trees. Even for small simple data sets, anything less than 500 trees is not nearly enough. $\endgroup$
    – Bogdanovist
    Aug 5, 2013 at 2:39
  • 2
    $\begingroup$ There are some minor caveats to 'more is always better' for the number of trees, but it's my understanding that you need gazjillions of trees before you start taking a performance hit. In my experience, as many trees as you have the CPU resources and patience to generate the better, albeit with diminishing returns once the OBB(ntrees) curve plateaus. $\endgroup$
    – Bogdanovist
    Aug 5, 2013 at 2:41
16
$\begingroup$

I think the answer is the max_features parameter: int, string or None, optional (default=”auto”) parameter. basically for this problem you should set it to None , so that each tree is built with all the inputs, since clearly you can't build a proper classifier using only a fraction of the cards ( default "auto" is selecting sqrt(nfeatures) inputs for each tree)

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ That was it! 95% accuracy with 50 trees and 600k training examples. $\endgroup$
    – Uwat
    Aug 5, 2013 at 16:35
  • 3
    $\begingroup$ Note that at this point you're hardly using a random forest, but as other answers have stated, it's not the ideal classifier for this exact problem. $\endgroup$
    – Richard Rast
    May 26, 2016 at 1:43
  • $\begingroup$ This should be the accepted answer! $\endgroup$
    – kjetil b halvorsen ♦
    Jan 30, 2021 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.