Recently I'm working with random forest algorithms, due to their easy to use. I always devide my set into train and test subsets, usually out of bag error for forest build on train dataset is higher (by more then 10%) then on test dataset. I wonder if it implicate overfitting or is it natural, should those two errors be equal ? If so I think I should choose parameters of forest (like maximum depth or maximum number of observations in termial node) to obtain similar values of errors.
I understand your question to be (correct me if I'm wrong) that:
- You are training a random forest (RF).
- You have randomly divided your data into train and test sets.
- The measured performance of the RF is obtained through cross-validation on your train set.
- You then take the RF produced from your train dataset and look at its performance on your test set.
- Sometimes your performance on the test set is better than the average performance obtained through cross-validation on the train set.
The following points are worth noting:
- The RF you are applying to the test set is trained on more data than the RFs used in cross-validation. Depending on how much data you have, we may expect this first RF to have better performance.
- The test estimate of performance is an estimate using one data point, the estimate of performance on your train set is the average of multiple data points. You don't have a sense of uncertainty on the test set's estimated performance (at least, not through my understanding of your procedure). You certainly may be in a situation where the estimated test performance is sometimes higher on the the estimated train performance, sometimes lower, but not statistically significant in its difference.