1
$\begingroup$

The common way to summarize results in cross-validation is by taking the average of results of all folds.
But in case there is a deviation in the results of individual folds, can we use the results of a single fold?

For example, the MSE of a 3 folds CV are 4, .556, and .03. Can I use the least MSE of .03 as the final result?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Can you provide a little more context, or an example to illustrate this? I'm not sure exactly what you mean. $\endgroup$
    – gung - Reinstate Monica
    Commented Mar 10, 2017 at 17:54
  • 1
    $\begingroup$ I added an example where there is a deviation in MSE. $\endgroup$
    – Ahmed Gad
    Commented Mar 10, 2017 at 17:57
  • 2
    $\begingroup$ Of course you can, but it will probably be a poor estimate of test mse $\endgroup$
    – Łukasz Grad
    Commented Mar 10, 2017 at 19:06
  • 1
    $\begingroup$ Thanks for that. Beside accepting that option of using a single fold to get the final result, can you please tell me the best option in such case? $\endgroup$
    – Ahmed Gad
    Commented Mar 10, 2017 at 19:08
  • 1
    $\begingroup$ You might try to investigate why there are such large differences. What is your sample size? Maybe there are outliers? ... $\endgroup$
    – kjetil b halvorsen ♦
    Commented Mar 10, 2017 at 19:31

1 Answer 1

Reset to default
3
$\begingroup$

the MSE of a 3 folds CV are 4, .556, and .03. Can I use the least MSE of .03 as the final result?

Picking the singe best split is going to give you a very optimistically biased estimate of generalization error, so don't do that.

There are (at least) 2 sources of variance in the results of CV splits:

  • As you are testing with different test cases, results are subject to a variance due to the finite number of tested cases. If this is too small, you'll have high variance (that variance is expected to decrease with the absolute number of independent cases tested in each fold).
  • If you have too few training samples (typically relative to the model complexity), models become unstable, i.e. the exchanging part of the training samples between any two folds leads to vastly different models.

You can find out which of the causes is dominant by examining results of repeated/iterated cross validation: as you have several predictions for exactly the same test case (one per repetition/iteration) those prediction differ only by model instability (See e.g. our paper Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6 ). In contrast, predictions of several cases by the same surrogate model differ only by variance due to test cases.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.