2
$\begingroup$

So I got my training set with 70% of my data called "train" / 30% "test"

I use ctree to get my decision tree model with something like this code below :

model_ctree <- ctree(response ~ x1 + .. xn , data = train) 

How can I apply this model to "test" and evaluate the model, use something like lift or gain chart or ROC; something that I would normally get from SAS miner?

I am new to R.

$\endgroup$
  • $\begingroup$ Dear JPC, welcome to cross validated. I voted to move your question to stackoverflow, because it is not asking about statistics but about programming (you already know you want an ROC as opposed to asking what ROCs are good for). $\endgroup$ – cbeleites Feb 6 '13 at 18:23
  • $\begingroup$ Also, please tell us what resources you used to look up what you are looking for, and get some introductions/tutorials on R. Many of them will tell you that R has help searching functions and packages that you can use to search e.g. for "ROC". Particularly, have a look at package "sos". Also, virtually any introduction that tells you how to fit a model a predictive model will also tell you how to use it to predict new cases. $\endgroup$ – cbeleites Feb 6 '13 at 18:23
  • $\begingroup$ I can't tell yet whether this question belongs on SO or here. Are you wondering which method (eg gain chart vs ROC) would be optimal, or just how to get R to perform that method? If the former, I think it should stay here; if both, I think it could go to either, based on your preferences. (I'm guessing you'd prefer here, since you asked it here.) We only need to migrate if it's just the latter. $\endgroup$ – gung Feb 6 '13 at 19:15
  • $\begingroup$ HI guys, thanks for the response first of all :) @gung : I do want some recommendation as far as what type of model evaluation I should use in this case. I will move this to SO. $\endgroup$ – JPC Feb 6 '13 at 20:18
  • 1
    $\begingroup$ Welcome to this site, JPC. Please, don't cross-post but rather ask a moderator (by flagging your post) to migrate your question. The duplicate on SO is unlikely to be well received as it stands, but see this discussion on our Meta. I agree with @gung that this question could well be on-topic here if emphasis was put on the statistical aspect of model evaluation rather than programming. $\endgroup$ – chl Feb 6 '13 at 22:07
7
$\begingroup$

Try this for class predictions:

pred <- predict(model_ctree, newdata=test)
library(caret)
confusionMatrix(pred, test$response)

Try this for class probabilities:

probs <- treeresponse(model_ctree, newdata=test)
pred <- do.call(rbind, pred)
summary(pred)

Try this for a roc curve:

library(ROCR)
roc_pred <- prediction(pred[,1], test$response)
plot(performance(roc_pred, measure="tpr", x.measure="fpr"), colorize=TRUE)

Try this for a lift curve:

plot(performance(roc_pred, measure="lift", x.measure="rpp"), colorize=TRUE)

Sensitivity/specificity curve and precision/recall curve:

plot(performance(roc_pred, measure="sens", x.measure="spec"), colorize=TRUE)
plot(performance(roc_pred, measure="prec", x.measure="rec"), colorize=TRUE)

More info:

?ctree
?confusionMatrix
?performance

Also, you should check out the caret package if you're building predictive models in R. It implements a number of out-of-sample evaluation schemes, including bootstrap sampling, cross-validation, and multiple train/test splits. caret is really nice because it provides a unified interface to all the models, so you don't have to remember, e.g., that treeresponse is the function to get class probabilities from a ctree model. Here's an example of using 10-fold cross-validation to evaluation your model, which is much better than a single train/test split:

model <- train(response ~ x1 + .. xn , data = train, method='ctree', tuneLength=10,
               trControl=trainControl(
                 method='cv', number=10, classProbs=TRUE, summaryFunction=twoClassSummary))
model
plot(model)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.