Applying random forest model to the data from which it is trained

Question

I am training a model using random forest in Python. The dataset is about 5 million entries with ~60 numerical features, and my target is to do a classification among 30 different groups.

When I apply the model to the data I used for training, it gives almost perfect prediction. That makes me feel like I am overfitting this.

In general, what is supposed to happen if one trains a random forest model with a dataset, then make predictions on the exactly same data? What are good and bad signs, and what shall one consider doing in any bad scenario?

Answer 1

It depends what you are classifying. It might just be a data-set which is very easy to classify since the classes are well separable. The only way to know for sure if you are overfitting is by comparing this training set performance to out-of-sample predictions.

Answer 2

You always want to prevent overfitting by checking the prediction on a test set, but where you know the results.

I suggest you try the following:

In Python, you can use the function train_test_split to split your data to train / test.

Then use GridSearchCV to tune your models parameters.

When you are done with tuning, fit the predictor to train data, predict on the test data and see the results. This should give you a hint how well your model will perform on new, unknown data.

You can check out my ipython notebook I made recently for a friend, where is coded how this can be done (+ some more stuff not relevant to the question, only the first 6 cells are relevant).