1
$\begingroup$

I have built a classification tree for factor variables with the rpart package and now I want to predict unseen data with it.

  1. How can I get a sense of whether the model is good at predicting unseen data or not ?

  2. Is it possible to represent graphically the data and the model in order to see the difference between them ?

$\endgroup$
1
$\begingroup$

First you need to define your measure of "good." It is always an absolute must to have some sort of a baseline model in place. Your classification tree may predict on a held out set with 0.9 accuracy, but this is less than impressive if a majority baseline model (classify everything as the majority class from the training data) also does this well.

Once you have defined this and performed your parameter tuning (I am guessing in your case it is mostly likely things like splitting criteria, tree complexity, etc.) via cross validation on 90% of the data you have available to you, apply the model to your remaining 10% of the data. If your model performs better on this data than your baseline, you can justify that your tree is doing well.

As for the second part of your question, I'm not really sure what you are looking for in terms of comparing the data to the model, but you might consider examining the class distribution across the nodes. Since you are using rpart package, you can use the rattle package to generate some decent looking visualizations like this:

Visualization of Iris dataset, because whatever

$\endgroup$
1
  • $\begingroup$ Thanks james. I have built a model with rpart() for 90% of my data and I plot the classification tree like yours with rattle. Now I want to use my classification tree on the 10% remaining data. How to use the model built on the 90% on the 10% new data ? and how to assess if the model is good at predicting that new data ? (predicted new data vs. actual new data) $\endgroup$
    – Synleb
    Jul 14 '15 at 22:40
-1
$\begingroup$
  1. In order to evaluate if the model or the prediction is "good" or not, you need to know the loss function. For example, if you are doing regression, you can use $f(y,\hat y)=(y-\hat y)^2$ as loss. For classification task, 0-1 loss (number of correct predictions) can be used.

  2. Depends on number of features and number of data point of your data, and the task you are doing (classification or regression) there are different ways to visualize.

Classification tree visualization example is given. Here is an example of visualizing regression trees (figures extracted from Wei-yin Loh's slides).

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.