3
$\begingroup$

If you use 10-fold cross validation to derive the error in, say, a C4.5 algorithm, then you are essentially building 10 separate trees on 90% of the data to test on 10% - 10 times. Which one of the 10 trees is representative? Won't they all be different?

For example - how does WEKA give me a C4.5 tree and a cross-validation error, but only one. I feel I must have fundamentally misunderstood this.

Thanks for any help

|cite|improve this question
$\endgroup$
6
$\begingroup$

Typically, you use the 10 cross-validated trees to estimate "out-of-sample" error, and then fit an 11th and final tree on the full dataset.

In theory, the error of the 11th tree on out-of-sample data should be similar to the out-of-sample error you estimated from the 10 cross-validated trees.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ @rosser no worries! I had the same question the first time I used caret in R to cross validate a model. $\endgroup$ – Zach Mar 5 '12 at 20:38
2
$\begingroup$

Yes, they will all be different, unless something surprising happens. The cross-validation is used to derive the error given the combination of algorithm choice and parameterization of that algorithm as applied to your problem; in this case, C4.5 as an algorithm and whatever parameterization you used for it. Its objective is, very roughly, to estimate how well your approach would work if used over and over again on similar data; if you did use it over and over again on similar data, you'd be building different trees with each new dataset. Consequently, there is no representative tree generated within the cross-validation loop; the tree you would use going forward would be constructed on the entire dataset.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Well said @jbowman. Unless all the trees turn out to have the same structure and give the same predictions, none of them is representative. And the typically poor cross-validating trees is why random forests, boosting, or bagging are more appropriate. I can't see that fitting one tree on the whole sample is an adequate summary, if the tree structure doesn't replicate over cross-validations. $\endgroup$ – Frank Harrell Mar 5 '12 at 21:15
  • $\begingroup$ @FrankHarrell - excellent points (+1). $\endgroup$ – jbowman Mar 5 '12 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.