Cross-Validation: Use results of a single fold The common way to summarize results in cross-validation is by taking the average of results of all folds.
But in case there is a deviation in the results of individual folds, can we use the results of a single fold?
For example, the MSE of a 3 folds CV are 4, .556, and .03. Can I use the least MSE of .03 as the final result?
 A: 
the MSE of a 3 folds CV are 4, .556, and .03. Can I use the least MSE of .03 as the final result?

Picking the singe best split is going to give you a very optimistically biased estimate of generalization error, so don't do that.
There are (at least) 2 sources of variance in the results of CV splits:


*

*As you are testing with different test cases, results are subject to a variance due to the finite number of tested cases. If this is too small, you'll have high variance (that variance is expected to decrease with the absolute number of independent cases tested in each fold).

*If you have too few training samples (typically relative to the model complexity), models become unstable, i.e. the exchanging part of the training samples between any two folds leads to vastly different models. 


You can find out which of the causes is dominant by examining results of repeated/iterated cross validation: as you have several predictions for exactly the same test case (one per repetition/iteration) those prediction differ only by model instability (See e.g. our paper Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6
).  In contrast, predictions of several cases by the same surrogate model differ only by variance due to test cases. 
