2
$\begingroup$

Set-up

I have a chemometric dataset with a hierarchical structure: 18 patients with ~250 spectra per patient. I want to compare some binary classifiers on this dataset which has 9 patients with and 9 without a disease.

For hyper-parameter optimisation I am performing leave-two-patients-out cross-validation (one with and one without the disease). I chose this because it seemed sensible to have positive and negative cases in the validation set. I'm simply using accuracy to compare performance.

The problem

A curious thing occurs with this set-up. There is a very high variance in the accuracy of a classifier depending on which two patients are held aside to assess the accuracy. The range of accuracy on the validation sets varies from 92% to as low as 12% in one instance. Even aside these two extremes, there is considerable variation in results.

My interpretations

1.) Perhaps there is naturally a significant degree of inter-patient variation in such samples making it difficult to generalise the performance to new patients. The sample size may be too small to capture this natural variation - i.e. it's not representative of the population.

2.) Perhaps there is a labelling error (especially the case with the 12% accuracy). I will ask, but it might be some time before an expert can have a look, so I want to be sure it's worth the effort first.

3.) I have heard of model instability, but I'm not sure I understand it. Is this a manifestation of it? Model instability implies to me that instability (i.e. high variance) is caused by the model - but would it not be the case that the instability is a feature of the data rather than the models? Or at least an interaction of the models and the data?

The classifiers I want to compare are PCA-LDA, SVM and a CNN. I plan to perform an external CV to assess which performs best (so the entire process will be nested CV). The instability is present, and roughly the same, for all these models.

4.) I've done something silly.

$\endgroup$
11
  • $\begingroup$ Can you please explain why you obtain an accuracy just from teh two samples left out? In LOO, the accuracy is computed over all symples cyclically left out. $\endgroup$
    – cdalitz
    Commented Oct 29, 2020 at 14:34
  • $\begingroup$ I obtain repeated measures of the accuracy, one for every pair of patients left out. From this I can obtain the average accuracy and its standard deviation. It is the variance of these accuracies that has me bemused. $\endgroup$
    – N Blake
    Commented Oct 29, 2020 at 15:36
  • $\begingroup$ I don’t see the benefit in leave two out vs leave one out. There isn’t need to ensure each fold has a positive and a negative case as each record eventually gets tested anyway in LOO $\endgroup$
    – astel
    Commented Oct 29, 2020 at 17:06
  • $\begingroup$ You only have k=9 CV measurements. As the variance of the mean value goes with $1/\sqrt{k}$, I would expect a large variance from such a small sample. $\endgroup$
    – cdalitz
    Commented Oct 30, 2020 at 14:10
  • $\begingroup$ @astel I suppose that is true so long as I'm using accuracy - if I were using sensitivity/specificity then the lack of pos or neg cases would be problematic? $\endgroup$
    – N Blake
    Commented Nov 2, 2020 at 12:31

1 Answer 1

1
$\begingroup$

Some issues to consider here:

First, the number of cases is very small, regardless of whether your sample is representative of the population. That leads to:

Second, your CV is necessarily something close to leave-one-out CV (in this case, one per group), which is known to have high variance. See Section 5.1.4 of An Introduction to Statistical Learning. With your approach, you only have a total of 81 models (9*9) among which to average.

Third, accuracy is a poor measure of model performance.

So try broader approaches for hyperparameter choices, like multiply repeated 5-fold CV (without necessarily limiting yourself to equal numbers of each class) or bootstrapping. During CV or bootstrapping, use a proper scoring rule like log-loss (equivalent to the logistic regression maximum-likelihood deviance criterion) or the Brier score (the classification equivalent of mean-square error) for evaluating performance.

You will still be limited by the small number of cases and potential un-representativeness of your sample. But those approaches probably have a better chance of working with your present data set.

$\endgroup$
3
  • $\begingroup$ The statement that LOO CV is known to have high variance is either wrong or at least not categorically correct. See stats.stackexchange.com/questions/61783/… $\endgroup$
    – astel
    Commented Oct 29, 2020 at 16:54
  • $\begingroup$ Lot's to consider, thanks. I've never come across things like the Brier score in the medical literature. I'll look into it and get back with what I've found. $\endgroup$
    – N Blake
    Commented Nov 2, 2020 at 12:41
  • $\begingroup$ @NBlake take a look at the resources provided by Frank Harrell, a well respected expert on biostatistics. He freely provides pdfs of course notes, and has written a comprehensive text on Regression Modeling Strategies. See the links from that page. Those should give you a good start on important statistical considerations that are too often glossed over in the biomedical literature. $\endgroup$
    – EdM
    Commented Nov 2, 2020 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.