I'm experimenting with random forests with scikit-learn and I'm getting great results of my training set, but relatively poor results on my test set...
Here is the problem (inspired from poker) which I'm trying to solve: Given player A's hole cards, player B's hole cards and a flop (3 cards), which player has the best hand? Mathematically, this is 14 inputs (7 cards -- one rank and one suit for each) and one output (0 or 1).
Here are some of my results so far:
Training set size: 600k, test set size: 120k, number of trees: 25 Success rate in training set: 99.975% Success rate in testing set: 90.05% Training set size: 400k, test set size: 80k, number of trees: 100 Success rate in training set: 100% Success rate in testing set: 89.7% Training set size: 600k, test set size: 120k, number of trees: 5 Success rate in training set: 98.685% Success rate in testing set: 85.69%
Here is the relevant code used:
from sklearn.ensemble import RandomForestClassifier Forest = RandomForestClassifier(n_estimators = 25) #n_estimator varies Forest = Forest.fit(inputs[:trainingSetSize],outputs[:trainingSetSize]) trainingOutputs = Forest.predict(inputs[:trainingSetSize]) testOutputs = Forest.predict(inputs[trainingSetSize:])
It appears that regardless of the number of trees used, performance on training set is much better than on test set, despite a relatively large training set and a reasonably small number of features...