I want to estimate a classifier accuracy on benchmark data. Data is not split into training and testing so I use 5-fold cross validation, using 80% of data as training and testing on 20%. Each test is repeated 20 times, so in total there are 100 runs (20 test runs * 5 tests on each fold) Accuracy is defined as number of correct predictions divided by number of records in a training data
I do not know how to calculate average accuracy and its standard deviation:
- Should results from each fold be averaged and then the stdev calculated on 20 samples?
- Should I calculate average and stdev on all 100 samples?
Another question is should STDEV or STDEVP function be used to calculate standard deviation, they are defined as follows:
STDEVP - Calculates standard deviation based on the entire population given as arguments.
STDEV - Estimates standard deviation based on a sample.