0
$\begingroup$

Given a binary classification problem and a model construction algorithm, suppose I run, as an example, 5 fold (stratified) cross-validation 4 times. I can think of these as 5x4 = 20 training and 20 holdout sets.

Can I estimate the accuracy of the classifier as follows?

  1. For every data instance, find the holdout sets where this data instance is present.

  2. Average the scores for this data instance over these holdout sets.

  3. Assign the prediction label to each data instance based on the size of this average score.

  4. Compute the accuracy as usual from 3.

Since each score is computed on holdout sets that are each completely disjoint from their corresponding training sets (no duplicates, no interpolation), this seems to me to be a valid way to measure the accuracy of the classifier - but I'm not 100% sure. Would it be a realistic estimate of the accuracy? For the particular data set I'm working on, this method seems to give better accuracy than simply averaging the accuracy over the 4 runs.

[edited slightly for clarity]

$\endgroup$
5
  • $\begingroup$ Can you elaborate on what youre doing in step 3? $\endgroup$
    – user20160
    Sep 20, 2019 at 11:57
  • $\begingroup$ Simply predicting the label that has the higher score. The scores may be calibrated beforehand if required. $\endgroup$
    – skm
    Sep 20, 2019 at 14:57
  • $\begingroup$ Realistic in what sense? I don't see any direct benefit for repeating the cross-validation x-amount of times. $\endgroup$
    – Scholar
    Sep 20, 2019 at 15:47
  • $\begingroup$ Does 'scores' refer to predicted class probabilities or something similar? i.e. some measure of how likely a point is to be a member of each class. This word can mean different things in different contexts. $\endgroup$
    – user20160
    Sep 20, 2019 at 15:51
  • $\begingroup$ Yes, here scores refer to class probabilities. $\endgroup$
    – skm
    Sep 21, 2019 at 13:53

1 Answer 1

1
$\begingroup$

Yes, you can do that: this will be the cross validation analogue to an out-of-bag error.

The model whose predictive performance this approximates is the ensemble model that uses aggregated predictions of aĺl 5 x 4 = 20 surrogate models.

For a reference, see e.g. our paper: Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6

$\endgroup$
2
  • $\begingroup$ Thank you very much for the reply and the link. Incidentally, "aggregate" is the word that I chose to use in my code. Currently I'm experimenting with fMRI data. Do you have a link to a preprint or some informal version of the article that you can share? $\endgroup$
    – skm
    Sep 23, 2019 at 7:25
  • $\begingroup$ @skm: of course! Please email me (see my profile) so I can send it back to you. $\endgroup$ Sep 23, 2019 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.