I am trying to find a good model for my data to predict new values. My idea was to combine a MARS and a CART model (averaging the outcomes of both models). This method fits the Training data well. Now I want to get some statistics which show how good this combined model is concerning prediction.
How can I estimate the predictive power in this case? Is cross-validation useful?
Boosting up till date is the most powerful way of combining CARTs. If by some way you think a model is only an estimate and should have a standard error, then model averaging is a good idea.
Note that you can always fit the training data as much as you want (over-fitting). And you will never judge a model on how it fits the training.
Therefore cross-validation bootstrapping and other out sample techniques are the only way to access the predictive performance of a model.
The performance measure use in assessing a model also plays an important rule. Predictive power is not often used, as power usually refers to hypothesis testing. Predictive performance is a better term to use.
