I have a dataset in a healthcare setting for which the task is to predict a binary outcome, I have done this using the Support Vector Machine algorithm in a 5-fold cross validation setup. The predictive validity is measured in Area Under Curve, in order to compare the result to existing literature. I have computed the AUC by averaging the AUCs over five folds. Then I computed the variance of AUC for each fold using the DeLong method [1], averaged the variances over the five folds, took the square root to obtain the Standard Deviation, and then divided by $\sqrt5$ to obtain the Standard Error of the mean.

  1. Is this a correct approach for estimating the Standard Error of the mean AUC?

Subsequently, I have obtained another dataset, that is very similar but from another healthcare provider. These datasets cannot be merged, but I have applied the same training/testing procedure (5-fold cv, SVM) to this dataset to obtain a similar prediction, and measured its AUC and SE as well. Then finally, I have applied both trained models to the other dataset, which results in an AUC and a SD using the DeLong method as well. Thus, I have two cross-validated predictions, and two predictions on an independent dataset.

  1. Is it possible to assess statistical differences between these four predictions (6 possible comparisons)? Can I for example assume a distribution for difference in AUCs (normal, t (dof=?), other)?

Any help would be much appreciated!

[1] DeLong, E. R., DeLong, D. M. & Clarke-Pearson, D. L. Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics (1988). doi:10.2307/2531595


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.