1
$\begingroup$

I have a large sales dataset and was intending to use a CART tree to predict the sales price of each item depending on lots of input factors such as the sales region etc. To achieve this I used the rpart package in R. I calculated the tree and pruned it according to this website which means pruning it back to the level of the smallest 10-fold cross validation error.

I tested the predictive power of the tree on unknown data (randomly splitting my dataset 80:20 over and over). The results confuse me. I calculated the MAPE of the predictions on the test set for both the pruned and the unpruned model. Contrary to my expectations the MAPE for the pruned tree is 0.379 while the MAPE for the unpruned tree is 0.233. For my dataset the difference between those two numbers is fairly large. I was expecting the pruned tree to perform better than the unpruned tree on unknown data.

Is this a common thing to happen? What are possible reasons?

Improve this question
$\endgroup$
5
  • $\begingroup$ what is MAPE? what is the size of your data set, how many predictors do you consider? of which type? $\endgroup$
    – Antoine
    Commented May 23, 2016 at 9:23
  • $\begingroup$ @Antoine MAPE is the Mean Absolute Percentage Error (en.wikipedia.org/wiki/Mean_absolute_percentage_error). The dataset consists of 1.4 million entries. I consider 13 predictors which are categorical and have between 5 and 361 values. As the predicted variable is continous (sales price) I obviously use a regression tree. $\endgroup$
    – sebsch88
    Commented May 23, 2016 at 9:34
  • $\begingroup$ What you observe could be caused by many things. Without further information it will be tough to identify the issue. I think it would help if you provided a minimal reproducible example (small sample of your data and code). $\endgroup$
    – Antoine
    Commented May 23, 2016 at 9:42
  • 1
    $\begingroup$ make sure all the settings are right for a regression tree $\endgroup$
    – ShainaR
    Commented May 23, 2016 at 17:07
  • $\begingroup$ CART is a weak learner. Why not use a random forest? It can substantially outperform, and it is robust so noise has a smaller than expected impact to prediction. $\endgroup$
    – EngrStudent
    Commented Dec 12, 2019 at 21:11

2 Answers 2

Reset to default
1
$\begingroup$

rpart uses a specific cost function which involves a complexity parameter mixed with a base cost function which could be infogain or accuracy. None of these functions optimize for mape. It is no surprise that when you optimize for some error function (aka. some specific form of errors) you get something different for other loss function (mape for your case). Different loss functions have minimum at different points, otherwise the idea of modeling errors through loss function would be meaningless.

All of this leaving aside that cart with pruning it is not the most robust way to get a classifier, less better than random forests, for example.

Improve this answer
$\endgroup$
0
$\begingroup$

This result might not be that surprising. By pruning you effectively increase the bias of your model while hoping to decrease variance. You state that you have a lot of features and a large dataset which could mean you end up underfitting your model when pruning.

Try to compare your error measure for test vs training sets using unprunned and pruned models. If you see high bias and low variance for the pruned model this would suggest underfitting and you might not need prunning.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ CART will not be competitive with other approaches with regard to predictive discrimination. In addition the tree will be volatile as revealed by bootstrapping. $\endgroup$
    – Frank Harrell
    Commented Sep 1, 2016 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.