I have a cross validation error higher than independent test set error (3-10% difference in error). It is not a training error but a cross-validation error using LOOCV. Does anyone have similar kind of situation. Could someone provide me some intuition behind it.
Two possibilities come immediately to my mind:
For classifiction: LOO always tests with a class that is underrepresented in training. Some classifiers take relative frequencies of cases into account - testing always with a class that is underrepresented can lead to pessimistic bias.
We observed this e.g. with PLS-DA and discuss it in this paper:
Beleites, C.; Baumgartner, R.; Bowman, C.; Somorjai, R.; Steiner, G.; Salzer, R. & Sowa, M. G. Variance reduction in estimating classification error using sparse datasets, Chemom Intell Lab Syst, 79, 91 - 100 (2005).
Having only a finite number of cases for testing means that there is random error on the test results - and the observed difference may just be a symptom of this. However, without knowing what figure of merit you are using (and whether we're talking about classification or regression), we cannot judge how likely this explanation is.
For typical classification error figures of merit (proportions such as accuracy etc.) you can calculate confidence intervals using a binomial distribution. See e.g. our paper:
Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33. DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323