2
$\begingroup$

I want to estimate a classifier accuracy on benchmark data. Data is not split into training and testing so I use 5-fold cross validation, using 80% of data as training and testing on 20%. Each test is repeated 20 times, so in total there are 100 runs (20 test runs * 5 tests on each fold) Accuracy is defined as number of correct predictions divided by number of records in a training data

I do not know how to calculate average accuracy and its standard deviation:

  • Should results from each fold be averaged and then the stdev calculated on 20 samples?

or

  • Should I calculate average and stdev on all 100 samples?

Another question is should STDEV or STDEVP function be used to calculate standard deviation, they are defined as follows:

  • STDEVP - Calculates standard deviation based on the entire population given as arguments.

  • STDEV - Estimates standard deviation based on a sample.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

How to calculate standard deviation depends on which standard deviation you need.

Your results are subject to (at least) 2 differenct sources of variance:

  • variance due to the actual sample you have (finite test set)
  • variance due to the differences in the surrogate models (model instability)

You can characterize the model instability variance by comparing the predictions for the same sample against that sample's mean over all runs.

For performance characteristics that are fractions of tested cases such as % correct, error rate, sensitivity and so on, you can calculate the variance due to the finite number of test cases as variance of a binomial distribution. You can also use the variance of your per-sample-loss for each of the surrogate models. But if you calculate variance of loss for a whole run, you already mix in the model instability variance of $k=5$ surrogate models.

(For none of these you have the whole population, so you need the degrees of freedom corrected for sampling.)

Improve this answer
$\endgroup$
2
  • $\begingroup$ If I understood correctly, you say that for model instability it's suggested to run the model using always the same sample, saving each result in a list, and then calculate standard deviation of that list? $\endgroup$
    – Jason Angel
    Commented May 2, 2020 at 22:27
  • $\begingroup$ @JasonAngel: you predict always the same sample, but the predictions are made by different surrogate models which were trained by slightly different training subsamples (the different runs use different assignments of the samples to the folds, so e.g. sample 3 is left out together with sample 1 in run 1 and toghether with sample 5 in run 2: any difference in the prediction must then be caused by traning with sample 5 instead of sample 3 [and possibly indeterministic parts of the training algorithm]). $\endgroup$
    – cbeleites
    Commented May 2, 2020 at 22:32
0
$\begingroup$

Usually you calculate the performance metrics over all repeats of all partitions, so in your case, over your 5x20 = 100 partitions. Be aware that by repeating the processing that is done on your 5 partitions, it makes sense to redo the partitioning itself as well. So the 5 partitions are likely not the same for the repeats (see e.g. the implementation of caret::train). The mean and standard deviation of the accuracy are then calculated on the vector of obtained accuracies, so in your case the 100 obtained accuracies from the 5x20 partitions.

PS: in case your classes are unbalanced, you might want to use something else than accuracy, as a bad performance on smaller classes could be hidden in the better performance on bigger classes.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.