# High variance across k-fold CV classification accuracy estimates

I'm tackling a binary classification task, data is class-balanced. I'm training a multi-layer perceptron on this data. To estimate accuracy on unseen samples, I decided to perform $$10$$-fold cross validation. However, estimated validation accuracies on the $$10$$ folds have high variance ($$\mu \simeq 63\%, \sigma \simeq 3\%$$).

My first reaction was suspecting this is due to the random initialization of the neural network and local minima. So I decided to repeat the training/validation for each fold, to see if the variance was caused by local minima. I was wrong. The model does a consistent job within the same fold, it appears that some folds are "easier" than others.

What does this tell me about the dataset? What can be done further to have a better estimate?

• Couple of questions here: Why use a shallow neural network with CV? This is old school. Did you try other learning algorithms, e.g. Random Forest or XGBoost? Did you have a look at accuracy metrics such as precision/recall per class? – Digio Feb 8 '19 at 10:23
• @Digio my data is very high-dimensional, and I will form an ensemble of the resulting networks after CV. – Adel Redjimi Feb 8 '19 at 10:53
• Out of curiosity, what made you follow this specific approach? – Digio Feb 8 '19 at 11:13

• for a (true) accuracy of $$p = 63\,\%$$, a standard deviation of the observed accuracy $$s_{\hat p}$$ of 3 % would be expected for testing with roughly 250 cases (using binomial distribution): $$s_{\hat p} = \sqrt{\frac{p (1-p)}{n_{tested}}}$$
As you compare observed accuracy across folds, that would be 250 cases in each fold. So if your data set isn't $$\gg$$ 2500 cases, you probably just ran into the fact that accuracy (and other figures of merit that are fractions of tested cases) has rather bad variance properties.