Multiple cross validation consistently performing worse than chance

Question

I am calculating accuracy results from a single feature using LDA. I am very curious why a certain feature consistently performs at less than chance when calculating accuracy using a stratified 10-fold cross validation technique. I have an equally distributed binary class and performing multiple xvals results in an accuracy that fluctuates fairly tightly around 0.3.

This means the training data is consistently setting a boundary which when tested against is basically the opposite of what was trained on.

I am performing this stratified 10-fold xval 10-20 times and I would expect this to fluctuate around 0.5, but it is fairly static around the 0.3 to 0.4 area.

Thanks for the help!

Answer 1

So the problem appears to be a small number of samples and the fact that its a horrible feature. The p-value is close to 1, so the classes are pretty much drawn from the same distribution. Using synthetic data drawn from the same distribution with small N, I show there is a bias below 0.5 when performing multiple xvals that seems to asymptotically converge to 0.5 as N increases.