In the following thesis http://arno.uvt.nl/show.cgi?fid=147278 the user compute the AUC standard deviation as measure of robustness.

Let's say I have run a repeated (10) 10-cross validation experiment with predictions implemented via a Markov chain model. As a measure of robustness, I want to compute the SD of the AUC across the runs/folders for the test set.

Intuitively, a relatively small standard deviation implies that the model produces stable results in distinguishing conversion from non-conversion.

The project is based on a 10x 10 cross-validated procedure, which as a consequence generate 100 AUCs.

Now, to derive the AUC's SD summarizing the model I understand the process should be (it is not well defined in the paper):

a. the SD is computed across folders based on the actual folders' AUC values (we derive 10 SDs, one for each repetition).

b. The SDs obtained at step b1 are then averaged across runs (it leaves us with 1SD).

however, as there is not so much literature on the topic I want to know If someone can validate the reasoning above

Any help or suggestion appreciated


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.