Given a binary classification problem and a model construction algorithm, suppose I run, as an example, 5 fold (stratified) cross-validation 4 times. I can think of these as 5x4 = 20 training and 20 holdout sets.
Can I estimate the accuracy of the classifier as follows?
For every data instance, find the holdout sets where this data instance is present.
Average the scores for this data instance over these holdout sets.
Assign the prediction label to each data instance based on the size of this average score.
Compute the accuracy as usual from 3.
Since each score is computed on holdout sets that are each completely disjoint from their corresponding training sets (no duplicates, no interpolation), this seems to me to be a valid way to measure the accuracy of the classifier - but I'm not 100% sure. Would it be a realistic estimate of the accuracy? For the particular data set I'm working on, this method seems to give better accuracy than simply averaging the accuracy over the 4 runs.
[edited slightly for clarity]