0
$\begingroup$

I'm tackling a binary classification task, data is class-balanced. I'm training a multi-layer perceptron on this data. To estimate accuracy on unseen samples, I decided to perform $10$-fold cross validation. However, estimated validation accuracies on the $10$ folds have high variance ($\mu \simeq 63\%, \sigma \simeq 3\%$).

My first reaction was suspecting this is due to the random initialization of the neural network and local minima. So I decided to repeat the training/validation for each fold, to see if the variance was caused by local minima. I was wrong. The model does a consistent job within the same fold, it appears that some folds are "easier" than others.

What does this tell me about the dataset? What can be done further to have a better estimate?

Improve this question
$\endgroup$
3
  • $\begingroup$ Couple of questions here: Why use a shallow neural network with CV? This is old school. Did you try other learning algorithms, e.g. Random Forest or XGBoost? Did you have a look at accuracy metrics such as precision/recall per class? $\endgroup$
    – Digio
    Commented Feb 8, 2019 at 10:23
  • $\begingroup$ @Digio my data is very high-dimensional, and I will form an ensemble of the resulting networks after CV. $\endgroup$
    – Adel Redjimi
    Commented Feb 8, 2019 at 10:53
  • $\begingroup$ Out of curiosity, what made you follow this specific approach? $\endgroup$
    – Digio
    Commented Feb 8, 2019 at 11:13

1 Answer 1

Reset to default
1
$\begingroup$

A few thoughts:

  • for a (true) accuracy of $p = 63\,\%$, a standard deviation of the observed accuracy $s_{\hat p}$ of 3 % would be expected for testing with roughly 250 cases (using binomial distribution): $$s_{\hat p} = \sqrt{\frac{p (1-p)}{n_{tested}}}$$

    As you compare observed accuracy across folds, that would be 250 cases in each fold. So if your data set isn't $\gg$ 2500 cases, you probably just ran into the fact that accuracy (and other figures of merit that are fractions of tested cases) has rather bad variance properties.

  • Of course, you can also accidentally run into easy or difficult train/test splits for the surrogate models.
    You can separate this instability wrt. exchanging training cases from the variance due to the tested cases above using iterated/repeated k-fold cross validation: repeating the 10-fold with different 10 splits allows you directly to compare predictions for the same case but predicted by models obtained from slightly different training sets. See e.g. our paper Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6
  • And, as your classifier isn't fully deterministic, that is another source of variance (which you've already ruled out).
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.