2
$\begingroup$

I have a dataframe, 'datas', with 200 observations and a series of columns (some numeric, dummy, etc) and a binary class variable to be predicted that is called "bad_econ." I would like to get the model to predict whether bad_econ = is the case (1) or not (0). Simple enough.

I wrote the follow code, testing rpart (I also tried Tree which yielded different model but same bad results afterwards):

frmla <- bad_econ ~ blah1 + blah2 + blah3 + [etc...]

fit = rpart(frmla, method='class', data=datas)

printcp(fit) # display the results
plotcp(fit) # visualize cross-validation results
summary(fit) # detailed summary of splits
fit

This works, and outputs some results but I am used to dealing with AUC + confusion matrix to assess model validity, not "residual mean deviance" nor "distribution of residuals" which are totally foreign to me, despite research.

So, I write the following, feeding my original data back through the model:

pred = predict(fit)
table(pred, datas$bad_econ)

I would think this would yield a small table that looks like this:

pred 0  1
  0  x  y
  1  z  a

But instead I get the following...

pred                 0  1
  0                  8  0
  0.2                4  1
  0.333333333333333  8  4
  0.4                6  4
  0.666666666666667  5 10
  0.714285714285714  2  5
  0.782608695652174  5 18
  0.8                1  4
  0.857142857142857  1  6
  0.928571428571429  1 13
  1                  0 93

Why is it passing so many fractional values when all the values in that column are binary and I've forced method = "class" on the model AND type="class" on the predict?

So, (a) what is going wrong here, (b) does anyone have any recommendations on how I can get what I'm actually looking for out of this (confusion matrix, AUC) and/or does anyone have an explanation of residual mean deviance that doesn't look like I need a M.S in Stats to swallow it?

e.g. How effective was this?

Residual mean deviance:  0.09359 = 17.13 / 183 
Distribution of residuals:
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-0.9286  0.0000  0.0000  0.0000  0.1429  0.8000

Sorry this is lengthy and thank you for any responses - looking forward to being more involved in the community on some things I actually have done before :)

$\endgroup$
  • 1
    $\begingroup$ It's ambiguous, but I think this may be more about understanding the model than the code. I think this may be on topic here. $\endgroup$ – gung - Reinstate Monica Jul 30 '15 at 11:32
  • $\begingroup$ Suspect this answer might be helpful: stats.stackexchange.com/questions/64551/… $\endgroup$ – conjugateprior Jul 30 '15 at 17:36
  • $\begingroup$ conjugateprior, I appreciate the link but following those directions is exactly how I got into the above issue. If you have anything more specific about the above, I welcome it. $\endgroup$ – Solutioneering Jul 31 '15 at 2:40
  • $\begingroup$ gung, if there are ways I can clarify what I'm after let me know, happy to make changes and thanks for commenting. $\endgroup$ – Solutioneering Jul 31 '15 at 2:40
  • $\begingroup$ Try as.factor (bad_econ) in your formula. I think you did not specify your 1 and 0 to be a factor. Simply using the method class might not be enough. $\endgroup$ – Jens Jul 31 '15 at 20:54
2
$\begingroup$

Predicting with an rpart object also includes the parameter Type. From the help:

If the rpart object is a classification tree, then the default is to return prob predictions, a matrix whose columns are the probability of the first, second, etc. class.

In your case it returns the probabilities of the classes, not the class itself. Use the following code to get the classes.

pred = predict(fit, type = "class")
table(pred, datas$bad_econ)

for more information on residual mean deviance check this post on cv

Edit: For measuring AUC and plot the ROC, you could use these commands:

library(ROCR)
rocrpred <- prediction(pred[,2], data$bad_econ)
# print AUC value
as.numeric(performance(rocrpred, "auc")@y.values)
# plots the ROC curve with colors where the splits are.
plot(performance(rocrpred, "tpr", "fpr"), colorize = TRUE) 

But read the help for the performance function. Because it can also create lift charts, Sensitivity/specificity plots, or Precision/recall graphs.

I very seldom look at the residual mean deviance. But I compare the confusion matrix of the prediction model with the confusion matrix of a baseline model. Same goes for the roc curve and the auc values. Those are easier to explain to non-stats people.

$\endgroup$
  • 1
    $\begingroup$ This answer is incomplete. The interest in this question on this site focuses on the part that asks "does anyone have an explanation of residual mean deviance." Could you add some remarks to address that? $\endgroup$ – whuber Jul 30 '15 at 13:30
  • $\begingroup$ phiver, thank you for this. Just to clarify, as for your comment "In your case it returns the probabilities of the classes, not the class itself." - there are only two classes and there are 200 rows - so what are the 11 rows that it's giving me with probabilities? Are those the end nodes? $\endgroup$ – Solutioneering Jul 31 '15 at 2:44
  • $\begingroup$ Thanks whuber, I have looked at the post that phiver linked to but I don't entirely get it. When I'm dealing with ROC/AUC there are pretty simple rules there, same could be said of accuracy and recall - is there really no straightforward guidelines about Residual Mean Deviance as a measure of the effectiveness of the model? If not, how might I measure this model's effectiveness more easily? $\endgroup$ – Solutioneering Jul 31 '15 at 2:46
  • $\begingroup$ I updated my answer a bit. I'm sure why you only see 11 rows. I would have to see a sample to exactly see what is going on. $\endgroup$ – phiver Jul 31 '15 at 8:07
  • $\begingroup$ Great update, thank you phiver. I will try this and if it doesn't sort I will upload a sample. Really appreciate the help. $\endgroup$ – Solutioneering Aug 1 '15 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.