# Correct setup for leave-one-subject-out cross-validation

I've got a question concerning leave-one-subject-out cross-validation of a classifier and correct outlier handling in this case.

Let's suppose I've got 5 subjects. Within each subject the features are standardised (mean subtracted and divided by standard deviation) missing values or outliers are represented by NaNs.

Now I'm doing an outer leave-one-subject-out cross-validation. In every outer fold a classifier is trained on the data of 4 subjects and tested on the fifth held out subject. NaNs are filled with non-informational values i. e. the mean of the training or test set respectively.

First question: Setting NaNs to the mean of the training or test set is ok?

Second question: Is it ok not to standardise the features within the training set? So I have then standardised for every subject but then do not standardise them over subjects in the training set?

Now within each outer fold I plan to tune a classifier's parameter with help of another cross-validation.

Third question: Does the inner cross-validation necessarily need to be leave-one-subject-out as well? I suppose, but can't really see the differences and implications of using another folding for the inner cross-validation.

Fourth question: For the inner fold the NaNs again need to be set to non-informational values (mean) of the corresponding training "subset"?

Setting NaNs to the mean of the training or test set is ok?

You need to define a procedure that you always follow. I see two valid options here:

• either use some value (e.g. mean) calculated for that subject (see also below)
• or some value calculated for the training set, basically a hyperparameter "value to be used for replacing NAs". This should not be calculated from the whole test set (independent testing also means that no parameters calculated from other test subjects should be used: the processing should not depend on the composition of the test set).

Edit:

Which method to follow (replace by value computed within subject or by value computed within training set) should IMHO be decided from the knowledge about the application and the data, we cannot tell you more than very general guidelines here.

Why not by value computed within test set?: That would mean that the value used to replace NAs in test subject A depends on whether subject A is tested together with subject B or subject C – which doesn't seem to be a desirable or sensible behaviour to me.

You may also want to look up "Imputation" which is the general term for techniques that fill in missing values.

Centering and scaling (standardization): if you have "external" (scientific) knowledge that suggests that a standardization within the subjects should take place, then go ahead with that. Whether this is sensible depends on your application and data, so we cannot answer this question. For a more general discussion of centering and standardization, see e.g. Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one? and When conducting multiple regression, when should you center your predictor variables & when should you standardize them?

Now within each outer fold I plan to tune a classifier's parameter with help of another cross-validation.

With 4 subjects you probably won't be able to compare classifiers anyways: the uncertainty due to having only 4 test cases is far too high. Can't you fix this parameter by experience with similar data?

To illustrate the problem: assume you observe 4 correct classifications out of 4 test subjects. That gives you a point estimate of 100% correct predictions. If you look at confidence interval calculations for this, you get e.g. for the Agresti-Coull method a 95% ci of 45 - 105% (obviously not very precise with the small sample size), Bayes method with uniform prior makes it 55 - 100%. In any case it means that even if you observe perfect test results, it is not quite clear whether you can claim that the model is actually better than guessing. As long as you do not need to fear that fixing the parameter beforehand will produce a model that is clearly worse than guessing, you anyways cannot measure improvements in the practically important range.
The situation may be less drastic if you optimize e.g. Brier score but with 4 subjects I'd suspect that you still do not reach the precision you need for the expected improvement during the optimization.

Edit: Unfortunately, while 20 subjects are far more than 4, from a classifier validation statistics point of view, 20 is still very few.

We recently concluded that if you need to stick with the frequently used proportions for characterizing your classifier, at least in our field you cannot expect to have a useful precision in the test results with less than 75 - 100 test subjects (in the denominator of the proportion!). Again, you may be better off if you can switch to e.g. Brier's score, and with a paired design for classifier comparison, but I'd call it lucky if that gains you a factor of 5 in sample size.

You can find our thoughts here: Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323

AFAIK, dealing with the random uncertainty on test results during classifier optimization is an unsolved problem. (If not, I'd be extremely interested to papers about the solution!)

So my recommendation would be to do a preliminary experiment/analysis at the end of which you try to estimate the random uncertainty on the comparison results. If these do not allow to optimize (which I'd unfortunately expect to be the outcome), report this result and argue that in consequence you do not have any choice at the moment but fixing the hyper-parameters to some sensible (though not optimized) value.

Does the inner cross-validation necessarily need to be leave-one-subject-out as well?

If you do inner cross validation it would be better to do it subject-wise as well: without this, you'll get overly optimistic inner results. Which would not a problem iff the bias were constant. However, it usually isn't and you have the additional problem that due to the random uncertainty together with the optimistic bias you may observe many models that seem to be perfect. Among these you cannot distinguish (after all, they all seem to be perfect) which can completely mess up the optimization.
Again, with so few subjects I'd avoid this inner optimization and fix the parameter.

• A. I just mentioned 5 to have a reference number when talking about "training the remaining 4 subjects" etc. In this particular case, there is data from 20 subjects. Sorry for the confusion this may have caused. B. About the two valid options for setting NaNs: I don't know if understand you correctly. I thought it might be a valid option to set in each fold the NaNs in the left out subject for this fold to the mean of this set, and the NaNs of the other data which is used to train the classifier in this fold to the mean of that data.So keeping training and test set in every fold independent. – user27424 Jun 28 '13 at 17:06
• C. Thanks for your pointers! I'll check the linked questions about standardisation and maybe come back with that again later. The question was less about if I should standardise data, but rather how - just within each subject or again for the whole cross-subject data. – user27424 Jun 28 '13 at 17:07
• @statikop: I'm not sure, but we may have been talking at cross-purposes about the replacement of the NAs. (I think you mean NA = not available as opposed to NaN = not a number [things like $\pm \infty$ or other results of calculations that do not yield a number]). Anyways, I tried to clarify my answer and also expanded on the 20 subjects a bit more in detail. – cbeleites Jun 30 '13 at 15:32
• Thanks for your edit, remarks and pointers to the paper. I'll look through them next week. At first sight: I think in my case there is no problem with having 20 subjects since the number of subjects is very different from the number of samples -- there are more than 1000 samples per subject. Let me know if I'm getting it wrong on the fly, or otherwise I will notice so next week while looking at it in detail. – user27424 Jun 30 '13 at 16:21
• @statikop: the effective sample size of a data set with 1000 samples $\times$ 20 subjects can be anywhere between 20 and 20000 - I frequently have data of similar layout where the effective sample size is far closer to 20 than to 20000. Look e.g. at the example data in the paper I linked. Whether or not this is the case with your data cannot be said from outside, you'll have to check this. In the end the question boils down to how much more similar are samples of the same subject compared to samples of another subject. – cbeleites Jun 30 '13 at 16:39