1
$\begingroup$

Suppose you are given a dataset with 4 attributes (F1, F2, F3, and F4). The class label is contained in attribute F4.

Now you build a random forest classification model and you test its performance using 10-fold cross-validation. For building the model you have used all four attributes (F1, F2, F3, and F4).

The precision and recall of your experiment are both close to 100%. Is there anything that went wrong? Would you obtain similar performance if you used a decision tree instead?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Uh, are you predicting f4 with f4?? $\endgroup$
    – generic_user
    Oct 27, 2017 at 16:52
  • 1
    $\begingroup$ Because if you are, then a decision tree will do EVEN BETTER than a random forest. $\endgroup$
    – generic_user
    Oct 27, 2017 at 17:02
  • 2
    $\begingroup$ Is this a question for homework/from a textbook? $\endgroup$
    – Firebug
    Oct 27, 2017 at 19:06
  • $\begingroup$ Random forest classifier is much less prone to overfitting compared to a single decision tree. If you pick a single decision tree and there is a large discrepancy between error on training set and validation set, you probably have overfitting. If they are close, perhaps you have linearly separable classes. $\endgroup$
    – Vladislavs Dovgalecs
    Oct 27, 2017 at 20:09

2 Answers 2

Reset to default
1
$\begingroup$

Is there anything that went wrong?

Taking your question literally, yes. You used feature F4 to predict F4 (or at the very least, something derived from F4). This is clearly invalid, and the most clear cut possibility of data leakage.

Would you obtain similar performance if you used a decision tree instead?

The problem here is with the human, not the algorithm. This would be invalid if you used a random forest, tree, gradient booster, regression, svm, neural network, anything. As long as the algorithm can draw a line of slope one to any given granularity, it will do so, and predict F4 perfectly from itself.

Improve this answer
$\endgroup$
0
$\begingroup$

Assuming you are not using F4 as both a class and predictor... Its possible you have very, very strong class separation. It isn't common, but it can happen, particularly if this strong separation is accompanied by a fairly small dataset. Assuming you have a binary class variable, try plotting the roc curve for each of your 3 predictor variables. This should help you determine if one of the predictors is carrying most of the predictive power.

Another thing to keep in mind is that random forests begin to look a lot like single decision trees when the number of predictor variables is small (with only 3 predictors, the mtry parameter can only take on a few values). So in this case, it is very likely RF and decision trees will perform equivalently, or nearly so.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.