1
$\begingroup$

Suppose you are given a dataset with 4 attributes (F1, F2, F3, and F4). The class label is contained in attribute F4.

Now you build a random forest classification model and you test its performance using 10-fold cross-validation. For building the model you have used all four attributes (F1, F2, F3, and F4).

The precision and recall of your experiment are both close to 100%. Is there anything that went wrong? Would you obtain similar performance if you used a decision tree instead?

$\endgroup$
4
  • 2
    $\begingroup$ Uh, are you predicting f4 with f4?? $\endgroup$ Commented Oct 27, 2017 at 16:52
  • 1
    $\begingroup$ Because if you are, then a decision tree will do EVEN BETTER than a random forest. $\endgroup$ Commented Oct 27, 2017 at 17:02
  • 2
    $\begingroup$ Is this a question for homework/from a textbook? $\endgroup$
    – Firebug
    Commented Oct 27, 2017 at 19:06
  • $\begingroup$ Random forest classifier is much less prone to overfitting compared to a single decision tree. If you pick a single decision tree and there is a large discrepancy between error on training set and validation set, you probably have overfitting. If they are close, perhaps you have linearly separable classes. $\endgroup$ Commented Oct 27, 2017 at 20:09

2 Answers 2

1
$\begingroup$

Is there anything that went wrong?

Taking your question literally, yes. You used feature F4 to predict F4 (or at the very least, something derived from F4). This is clearly invalid, and the most clear cut possibility of data leakage.

Would you obtain similar performance if you used a decision tree instead?

The problem here is with the human, not the algorithm. This would be invalid if you used a random forest, tree, gradient booster, regression, svm, neural network, anything. As long as the algorithm can draw a line of slope one to any given granularity, it will do so, and predict F4 perfectly from itself.

$\endgroup$
0
$\begingroup$

Assuming you are not using F4 as both a class and predictor... Its possible you have very, very strong class separation. It isn't common, but it can happen, particularly if this strong separation is accompanied by a fairly small dataset. Assuming you have a binary class variable, try plotting the roc curve for each of your 3 predictor variables. This should help you determine if one of the predictors is carrying most of the predictive power.

Another thing to keep in mind is that random forests begin to look a lot like single decision trees when the number of predictor variables is small (with only 3 predictors, the mtry parameter can only take on a few values). So in this case, it is very likely RF and decision trees will perform equivalently, or nearly so.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.