Let's say that I'm trying to predict, based on a total of 10 physical features (height, weight, etc..), whether an individual is male or female. The population size is 150, so I have a 150x10 data matrix. I build a decision tree using the rpart package , and get a 80% hindsight accuracy for both males and females. Encouraged, I proceed to cross-validate via leave-50-out: randomly selecting 100 individuals to act as the training set for the decision tree and 50 individuals to act as the testing set. The prediction accuracy is saved as a two column vector (pred. accuracy for males, pred. accuracy for females). I repeat this 1000 times, and plot the resulting 1000x2 matrix. I do not know what to make of the resulting pattern (attached also a plot of 10,000 iterations so that the pattern I'm talking about can be more easily seen). Is this simply a case of some bias in the sampling function combined with poor predictive ability of the model?
Edit: A plot for 10k iterations, colored based on the amount of males in the test subset. (Edit #2 - prettyfied via ggplot2)
Edit 3 : a density plot of the results