2
$\begingroup$

I use regression trees (R package rpart) in my statistical analysis, and have received a critical comment that this method amounts to a "hunting expedition" that will always produce a result ("statistical creep"), but not necessarily one that helps in the answering the research question. So the commenter appears to be worried about the danger of overfitting. My options appear to be to either convince the reviewer that this problem is manageable or to choose a different method.

My questions are:

  1. Does pruning adequately handle the danger of overfitting? Or should I use alternative methods?
  2. Are regression trees more prone to overfitting or a "hunting expedition" than a normal mixed effects regression models?

Background:

In case this might be useful to answer the question, here's the background to my analysis. I'm investigating whether two phonemes differ acoustically between two dialects, and what other variables determine their acoustic realisation (or "pronunciation"). I have three dependent interval-scaled variables, and have computed three different regression trees for these. My independent variables are all categorical (dialect, preceding and following phonological environment, speaking style). I do not use mixed effects regression models because I assume there might be three-way interactions (and even more complex ones) between the variables, and these are difficult to handle and interpret with mixed effects regression.

$\endgroup$
1
$\begingroup$

Overfitting is a concern with trees, and pruning can certainly help avoid that. See The Elements of Statistical Learning, section 9.2, for a good discussion of the concern and a recommendation on pruning strategy.

That being said, in practice a Random Forest is just as easy to implement, will produce better results, and will be nearly impossible to overfit, so you may just want to go that route.

$\endgroup$
  • 2
    $\begingroup$ In addition to the pruning strategies employed in "classic" regression trees (especially cost-complexity pruning), one can also use so-called pre-pruning based on significance tests leading to an unbiased tree selection. This is done for example in ctree() and lmtree() in package partykit. The latter can also be combined with post-pruning based on information criteria. $\endgroup$ – Achim Zeileis Aug 17 '15 at 23:33
  • $\begingroup$ Many thanks for your useful answer +1 AFAICS, random forests only allow me to determine the importance of the IVs, right? Or can I actually use random forests to determine exactly how the IVs influence the DV? $\endgroup$ – robert Aug 18 '15 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.