I am cross validating a classification tree and am able to plot the number of observations misclassed by different sizes of trees. My question is, what does it mean to return "the" number misclassed, for a given tree size, when there were k different runs of that sized tree (where each of the k runs has a presumably different number misclassed)?
Several texts that I have read say that it is the average over the k folds that should be returned for each size of tree, but I do not think this is what I am getting since the numbers I see plotted for the "number misclassed" are always perfect integers.
Here is an example:
library('ISLR') attach(Carseats) High = cut(Sales, c(-Inf, 8, Inf), labels=c("Small", "Large")) Carseats = data.frame(Carseats, High) set.seed(2) train = sample(1:nrow(Carseats), 200) library('tree') tr0 = tree(High ~ . -Sales, data=Carseats, subset=train) set.seed(3) tr0.cv = cv.tree(tr0, FUN=prune.misclass) plot.tree.sequence(tr0.cv) tr0.cv$dev # 55 55 53 52 50 56 69 65 80 # These don't look like averages across k folds
From my example, here are the different tree sizes and their corresponding dev value (here, meaning number misclassed)
tr0.cv$size  19 17 14 13 9 7 3 2 1 tr0.cv$dev  55 55 53 52 50 56 69 65 80
So we have for a tree of size 19, the number misclassed is 55. So is this saying that across the 10 runs, the sum total of all misclasses was 55? Thus about 5.5 misclasses on average, for each individual tree fit of size 19?
This seems suspect, because when I do fit a tree of size 19, I see the number misclassed as 21.
tr0.Prune19 = prune.misclass(tr0, best=19) summary(tr0.Prune19) Classification tree: tree(formula = High ~ . - Sales, data = Carseats, subset = train) Number of terminal nodes: 19 Residual mean deviance: 0.4282 = 77.51 / 181 Misclassification error rate: 0.105 = 21 / 200
Doing this 10 times I would expect the total sum of number misclassed to be roughly 200, which is quite different from the reported 55. Granted, I know across 10 different runs there will be some variability in the number misclassed, but this sounds like too large of a discrepancy. Am I missing something?