0
$\begingroup$

Given a binary classification problem and a model construction algorithm, suppose I run, as an example, 5 fold (stratified) cross-validation 4 times. I can think of these as 5x4 = 20 training and 20 holdout sets.

Can I estimate the accuracy of the classifier as follows?

  1. For every data instance, find the holdout sets where this data instance is present.

  2. Average the scores for this data instance over these holdout sets.

  3. Assign the prediction label to each data instance based on the size of this average score.

  4. Compute the accuracy as usual from 3.

Since each score is computed on holdout sets that are each completely disjoint from their corresponding training sets (no duplicates, no interpolation), this seems to me to be a valid way to measure the accuracy of the classifier - but I'm not 100% sure. Would it be a realistic estimate of the accuracy? For the particular data set I'm working on, this method seems to give better accuracy than simply averaging the accuracy over the 4 runs.

[edited slightly for clarity]

Improve this question
$\endgroup$
5
  • $\begingroup$ Can you elaborate on what youre doing in step 3? $\endgroup$ – user20160 Sep 20 '19 at 11:57
  • $\begingroup$ Simply predicting the label that has the higher score. The scores may be calibrated beforehand if required. $\endgroup$ – skm Sep 20 '19 at 14:57
  • $\begingroup$ Realistic in what sense? I don't see any direct benefit for repeating the cross-validation x-amount of times. $\endgroup$ – Scholar Sep 20 '19 at 15:47
  • $\begingroup$ Does 'scores' refer to predicted class probabilities or something similar? i.e. some measure of how likely a point is to be a member of each class. This word can mean different things in different contexts. $\endgroup$ – user20160 Sep 20 '19 at 15:51
  • $\begingroup$ Yes, here scores refer to class probabilities. $\endgroup$ – skm Sep 21 '19 at 13:53
1
$\begingroup$

Yes, you can do that: this will be the cross validation analogue to an out-of-bag error.

The model whose predictive performance this approximates is the ensemble model that uses aggregated predictions of aĺl 5 x 4 = 20 surrogate models.

For a reference, see e.g. our paper: Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you very much for the reply and the link. Incidentally, "aggregate" is the word that I chose to use in my code. Currently I'm experimenting with fMRI data. Do you have a link to a preprint or some informal version of the article that you can share? $\endgroup$ – skm Sep 23 '19 at 7:25
  • $\begingroup$ @skm: of course! Please email me (see my profile) so I can send it back to you. $\endgroup$ – cbeleites unhappy with SX Sep 23 '19 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.