0
$\begingroup$

I am trying to solve the same classification problem with the R packages rpart and partykit. I would have expected better results from partykit::ctree as it seems to be the more 'sophisticated' method; however, my results are slightly worse.

I have the theory that this is because ctree uses the p-value of some statistical test for determining splits, and my dataset is relatively large (several million rows), and therefore I get to many significant p-values. Could this be the problem? And can anyone recommend and explains which settings to use for ctree (and or rpart) when working with large datasets?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The pre-pruning strategy based on p-values from significance tests is most useful for sample sizes for which significance tests work well. However, for very large data sets with millions of observations (as in your case) commonly used significance levels such as 5% are typically too large. In ctree() this can lead to some overfitting. To prevent this you can:

  • Use a much lower significance level, possibly selecting the best significance level by cross-validation (or similar).

  • Add a post-pruning step, e.g., based on cost-complexity pruning (as in rpart) or information criteria.

The former can be done relatively easily "by hand". The latter needs a little bit more work. The lmtree() and glmtree() functions in partykit support post-pruning based on AIC or BIC.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.