0
$\begingroup$

I use the package rpartto model a classification/regression tree. I have the variables $x,y,s$ where $x$ is in $\{-1,1\}$, y is continuous in $[0,1]$ and $s$ is a factor with 3 levels.

I use 

fit <- rpart(x~y+s, data=data, method="class")

The final model makes perfect sense, I can plot it using fancyRpartPlot(fit).

How can I quantify the confidence of the decisions in the tree? There is the function rsq.rpart and it gives the value "CP" and "rel error" and "xerror" and "xstd". How can I construct something like $R^2$ from this? As $y$ is like a factor $R^2$ will not be the right object I assume.

I would appreciate if you have a link to an intuitive introduction to this approach and analytics of it - if possible in my setting.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I think you want the confusion matrix . $\endgroup$
    – meh
    Apr 20, 2015 at 14:40

1 Answer 1

Reset to default
1
$\begingroup$

The "rel error" is $1−R^2$ so you can get $R^2$ with some arithmetic. You mention that you think $R^2$ is not the right object to use but it's perfectly fine if you just remember what it means. That is, $R^2$ has all the same problems in CART that it has in linear regression - it measures how well your model fits the data used to model it and so it does not account for over-fitting. If you want to assess the strength of your classifier to new data, you will have to use something like cross-validation where you asses how your classifier performs on entirely new data that wasn't used to build the classifier.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I worked with the confusion matrix and it turns out that there is an in-sample accuracy $#(correctly classified)/n$ of roughly $66\%$ but "rel error" gives 0.75 at the last split ($nsplit=3$) which would mean that $R^2\approx 0.25$ how does this fit together? Thanks! $\endgroup$
    – Richi W
    Apr 20, 2015 at 15:09
  • $\begingroup$ @Richard in sample accuracy is not the same relative error. $\endgroup$
    – TrynnaDoStat
    Apr 23, 2015 at 0:11
  • $\begingroup$ I know that it is not the same, but should not in sample accuracy relatively high relate to relative error relatively small? $\endgroup$
    – Richi W
    Apr 23, 2015 at 12:33
  • $\begingroup$ @Richard I'm a little confused by your comments. I would say that the in-sample accuracy and the relative error are giving fairly consistent stories. That is, one is not telling you the classification is doing fantastic while the other is telling you it's poor. $\endgroup$
    – TrynnaDoStat
    Apr 23, 2015 at 13:19
  • $\begingroup$ What I mean is that if accuracy is $66\%$ (I know the precise definition) is it then plausible to have a relative error of $75\%$? Do you know the precise definition of the relative error in this context? Thanks! $\endgroup$
    – Richi W
    Apr 23, 2015 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.