6
$\begingroup$

I am cross validating a classification tree and am able to plot the number of observations misclassed by different sizes of trees. My question is, what does it mean to return "the" number misclassed, for a given tree size, when there were k different runs of that sized tree (where each of the k runs has a presumably different number misclassed)?

Several texts that I have read say that it is the average over the k folds that should be returned for each size of tree, but I do not think this is what I am getting since the numbers I see plotted for the "number misclassed" are always perfect integers.

Here is an example:

library('ISLR')
attach(Carseats)

High = cut(Sales, c(-Inf, 8, Inf), labels=c("Small", "Large"))
Carseats = data.frame(Carseats, High)

set.seed(2)
train = sample(1:nrow(Carseats), 200)

library('tree')
tr0 = tree(High ~ . -Sales, data=Carseats, subset=train)

set.seed(3)
tr0.cv = cv.tree(tr0, FUN=prune.misclass)
plot.tree.sequence(tr0.cv)

tr0.cv$dev 
#[1] 55 55 53 52 50 56 69 65 80
# These don't look like averages across k folds

Update

From my example, here are the different tree sizes and their corresponding dev value (here, meaning number misclassed)

tr0.cv$size
[1] 19 17 14 13  9  7  3  2  1
tr0.cv$dev
[1] 55 55 53 52 50 56 69 65 80

So we have for a tree of size 19, the number misclassed is 55. So is this saying that across the 10 runs, the sum total of all misclasses was 55? Thus about 5.5 misclasses on average, for each individual tree fit of size 19?

This seems suspect, because when I do fit a tree of size 19, I see the number misclassed as 21.

tr0.Prune19 = prune.misclass(tr0, best=19)
summary(tr0.Prune19)

Classification tree:
tree(formula = High ~ . - Sales, data = Carseats, subset = train)
Number of terminal nodes:  19 
Residual mean deviance:  0.4282 = 77.51 / 181 
Misclassification error rate: 0.105 = 21 / 200 

Doing this 10 times I would expect the total sum of number misclassed to be roughly 200, which is quite different from the reported 55. Granted, I know across 10 different runs there will be some variability in the number misclassed, but this sounds like too large of a discrepancy. Am I missing something?

$\endgroup$
  • 1
    $\begingroup$ Learning and applying a tree of size 19 causes overfitting. That is why the 21 is lower than the 55 from cross validation. $\endgroup$ – Pieter May 19 '16 at 15:38
1
$\begingroup$

It returns the sum of deviances from each of the 10 fits, for a range of complexity parameters.

from reference Manual... "A copy of FUN applied to object, with component dev replaced by the cross-validated results from the sum of the dev components of each fit."

from code...

cvdev <- 0
for (i in unique(rand)) {
    tlearn <- tree(model = m[rand != i, , drop = FALSE])
    plearn <- do.call(FUN, c(list(tlearn, newdata = m[rand == 
        i, , drop = FALSE], k = init$k), extras))
    cvdev <- cvdev + plearn$dev
}

Notice the plearn$dev is summed across folds.

$\endgroup$
  • $\begingroup$ Thanks that would explain my result. I have one other concern though which I have written as an update to my original question. $\endgroup$ – moof Nov 26 '13 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.