I have a dataset in a healthcare setting for which the task is to predict a binary outcome, I have done this using the Support Vector Machine algorithm in a 5-fold cross validation setup. The predictive validity is measured in Area Under Curve, in order to compare the result to existing literature. I have computed the AUC by averaging the AUCs over five folds. Then I computed the variance of AUC for each fold using the DeLong method [1], averaged the variances over the five folds, took the square root to obtain the Standard Deviation, and then divided by $\sqrt5$ to obtain the Standard Error of the mean.

  1. Is this a correct approach for estimating the Standard Error of the mean AUC?

Subsequently, I have obtained another dataset, that is very similar but from another healthcare provider. These datasets cannot be merged, but I have applied the same training/testing procedure (5-fold cv, SVM) to this dataset to obtain a similar prediction, and measured its AUC and SE as well. Then finally, I have applied both trained models to the other dataset, which results in an AUC and a SD using the DeLong method as well. Thus, I have two cross-validated predictions, and two predictions on an independent dataset.

  1. Is it possible to assess statistical differences between these four predictions (6 possible comparisons)? Can I for example assume a distribution for difference in AUCs (normal, t (dof=?), other)?

Any help would be much appreciated!

[1] DeLong, E. R., DeLong, D. M. & Clarke-Pearson, D. L. Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics (1988). doi:10.2307/2531595


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.