# A strange pattern of cross-validation results

Let's say that I'm trying to predict, based on a total of 10 physical features (height, weight, etc..), whether an individual is male or female. The population size is 150, so I have a 150x10 data matrix. I build a decision tree using the rpart package , and get a 80% hindsight accuracy for both males and females. Encouraged, I proceed to cross-validate via leave-50-out: randomly selecting 100 individuals to act as the training set for the decision tree and 50 individuals to act as the testing set. The prediction accuracy is saved as a two column vector (pred. accuracy for males, pred. accuracy for females). I repeat this 1000 times, and plot the resulting 1000x2 matrix. I do not know what to make of the resulting pattern (attached also a plot of 10,000 iterations so that the pattern I'm talking about can be more easily seen). Is this simply a case of some bias in the sampling function combined with poor predictive ability of the model?  Edit: A plot for 10k iterations, colored based on the amount of males in the test subset. (Edit #2 - prettyfied via ggplot2) Edit 3 : a density plot of the results • What part of the plot do you find to be strange? I see several possibilities and am unsure what to express. What I think you might find interesting is to save how many of the 50 people left out were male and color your plot by that. This should give rise to some nice patterns. I would also be interested in seeing that plot.
– Erik
Sep 7 '12 at 12:55
• The image is a bit compressed horizontally when shown on the site, but if you save and open it in an image editor you will see the the points form line that remind me of a simple differential equations' field lines (look at the area around (0.5,0.5) ) Sep 7 '12 at 13:03
• Ok, this is what I suspected what you meant. This is basically the result of what I wrote. Let's say for example you have 20 males and 30 females, then your male accuracy will be some multiple of 5% and your female accuracy some multiple of 3.33%. The interplay between the number of males and the possible accuracies will result in these lines. As I said coloring it should make it clearer.
– Erik
Sep 7 '12 at 13:08
• If by hindsight accuracy oyu mean the resubstitution estimate of error rate, it is probably at least partially due to the well-known large small sample optimistic bias of the error rates. The true classification error rates are probably a lot higher than 20%. Sep 7 '12 at 13:42
• Yes, Michael, I'm aware of that. As you can see in the cross-validation plots, it seems that my classifier is about as good as flipping a coin. Sep 7 '12 at 13:44

Ok, I will try to complete my comments to make it an answer. The patterns you see is because your procedure can only give a discrete set of answers.

If you hold the number of males in your test set constant, all possible results will lie on a grid with a density of $1/\#\text{male}$ on the male accuracy axis and $1/\#\text{female}$ on the female accuracy axis. So this will give you straight lines.

The curves you also notice are slightly more subtle. Look at this example: Let's say you have have $m$ males, $f$ females and $m_k$ correctly classified males and $f_k$ correctly classified fk females. You get the point $(m_k/m,f_k/f)$. Note that $f = 50 - m$. If you change $m$ while leaving $m_k$ and $f_k$ constant ($m_k < m$ and $f_k < 50 -m$ being necessary!) you get a differential equation like relationship between $m_k/m$ and $f_k/f$.

There will also probably be curves with correspond to the the number of males and the number of either correctly classified females, males or both changing by 1.

I would also like to note that your plot may not give an accurate impression, because many of the points will be there multiple times while others will be there just once. Performance might therefore be better (or worse) than it seems to be.

You could also look at the plot of all possible results to help you visualize this:

output <- NULL
final.frame.list <- vector("list", 49)

for (i in 1:49)
{
next.output <- NULL
for (j in 0:i)
{
acc <- j/i
acc2 <- seq(0,1, b=1/(50-i))
next.output <- rbind(next.output, data.frame(m.acc = acc, f.acc = acc2))
}
final.frame.list[[i]] <- next.output
}
final.frame <- do.call(rbind, final.frame.list)
plot(final.frame)

• Thank you for explaining the patterns, Eric. Your comment regarding the duplicate points prompted me to create a density plot. It looks as if the cloud has a certain bias towards the 'good classifier' neighbourhood. Are you familiar with a way to quantify such a bias and to say whether it's statistically significant? Sep 7 '12 at 14:48
• @NoamN.Kremen Simply draw a boxplot of the overall accuracy and check (or test) if it is better than 50%.
– user88
Sep 7 '12 at 14:51
• There is a large literature about adjusting for classification bias using the bootstrap. I think the bootstrap can also be used to test whether or not a specific error rate estimate for a specific classifier has a statistically significant bias. As an aside the first dotted blue colored graph looks like a finger print with a grid appearing on it. Sep 7 '12 at 16:09
• Thanks mbq, boxplot shows overall accuracy is somewhere beween 0.55 and 0.8 for 75% percent of the points, but in some cases this is due to a high male accuracy combined with a <0.5 female accuracy. Given that my dataset is split 40% female, 60% male this is not very surprising. Thank you, Michael Chernick , I'll read up on your suggestions. Sep 8 '12 at 11:06