A strange pattern of cross-validation results Let's say that I'm trying to predict, based on a total of 10 physical features (height, weight, etc..), whether an individual is male or female. The population size is 150, so I have a 150x10 data matrix. I build a decision tree using the rpart package , and get a 80% hindsight accuracy for both males and females. Encouraged, I proceed to cross-validate via leave-50-out: randomly selecting 100 individuals to act as the training set for the decision tree and 50 individuals to act as the testing set. The prediction accuracy is saved as a two column vector (pred. accuracy for males, pred. accuracy for females). I repeat this 1000 times, and plot the resulting  1000x2 matrix. I do not know what to make of the resulting pattern (attached also a plot of 10,000 iterations so that the pattern I'm talking about can be more easily seen). Is this simply a case of some bias in the sampling function combined with poor predictive ability of the model?


Edit: A plot for 10k iterations, colored based on the amount of males in the test subset. (Edit #2 - prettyfied via ggplot2)

Edit 3 : a density plot of the results

 A: Ok, I will try to complete my comments to make it an answer. The patterns you see is because your procedure can only give a discrete set of answers. 
If you hold the number of males in your test set constant, all possible results will lie on a grid with a density of $1/\#\text{male}$ on the male accuracy axis and $1/\#\text{female}$ on the female accuracy axis. So this will give you straight lines. 
The curves you also notice are slightly more subtle. Look at this example: Let's say you have have $m$ males, $f$ females and $m_k$ correctly classified males and $f_k$ correctly classified fk females. You get the point $(m_k/m,f_k/f)$. Note that $f = 50 - m$. If you change $m$ while leaving $m_k$ and $f_k$ constant ($m_k < m$ and $f_k < 50 -m$ being necessary!) you get a differential equation like relationship between $m_k/m$ and $f_k/f$.  
There will also probably be curves with correspond to the the number of males and the number of either correctly classified females, males or both changing by 1.    
I would also like to note that your plot may not give an accurate impression, because many of the points will be there multiple times while others will be there just once. Performance might therefore be better (or worse) than it seems to be.
You could also look at the plot of all possible results to help you visualize this:
output <- NULL
final.frame.list <- vector("list", 49)

for (i in 1:49)
{
    next.output <- NULL
    for (j in 0:i)
    {
        acc <- j/i
        acc2 <- seq(0,1, b=1/(50-i))
        next.output <- rbind(next.output, data.frame(m.acc = acc, f.acc = acc2))
    }
    final.frame.list[[i]] <- next.output
}
final.frame <- do.call(rbind, final.frame.list)
plot(final.frame)

