Avoiding overfitting with linear regression trees

Question

I use regression trees (R package rpart) in my statistical analysis, and have received a critical comment that this method amounts to a "hunting expedition" that will always produce a result ("statistical creep"), but not necessarily one that helps in the answering the research question. So the commenter appears to be worried about the danger of overfitting. My options appear to be to either convince the reviewer that this problem is manageable or to choose a different method.

My questions are:

1. Does pruning adequately handle the danger of overfitting? Or should I use alternative methods?
2. Are regression trees more prone to overfitting or a "hunting expedition" than a normal mixed effects regression models?

Background:

In case this might be useful to answer the question, here's the background to my analysis. I'm investigating whether two phonemes differ acoustically between two dialects, and what other variables determine their acoustic realisation (or "pronunciation"). I have three dependent interval-scaled variables, and have computed three different regression trees for these. My independent variables are all categorical (dialect, preceding and following phonological environment, speaking style). I do not use mixed effects regression models because I assume there might be three-way interactions (and even more complex ones) between the variables, and these are difficult to handle and interpret with mixed effects regression.

Answer 1

Overfitting is a concern with trees, and pruning can certainly help avoid that. See The Elements of Statistical Learning, section 9.2, for a good discussion of the concern and a recommendation on pruning strategy.

That being said, in practice a Random Forest is just as easy to implement, will produce better results, and will be nearly impossible to overfit, so you may just want to go that route.