One model performs better than the other. How to measure if it is statistically significant? So, let's say that I train two models on the same dataset. I run the experiment once and I get the following results:


*

*Using a Neural Network I get an AUC ROC of 0.941.

*Using Random Forest I get an AUC ROC of 0.947.


However, both algorithms have some random processes inside, and therefore if I would run the experiment again, the results may vary slightly.
My question is: how should I measure/evaluate the statistical significance of this improvent? When is it safe to claim that one algorithm is doing better than the other?
Also, I have read a lot of Machine Learning papers where they do not measure the statistical significance between the results obtained by the proposed model and the baseline model. So I guess that when the difference is big enough, there is no need to evaluate its statistical significance? If so, when is the difference considered big enough? I'd love to see what the community thinks about this issue.
Thanks a lot!
 A: "Not statistically significant" means that the outcome you observed would be likely to happen (typically meaning with a probability > 5%) under the hypothesis that the two methods are equally good (null hypothesis).
So the problem is to figure out how likely it would be to observing the result under that hypothesis. In this case it could be due to:


*

*that particular dataset just favors the random forest algorithm

*you got lucky with the random processes in the algorithms


For the second issue you can certainly run you experiment multiple times, and see if the random forest method consistently outperforms the other.
If you have a large enough test dataset you could split it randomly and see if your results are consistent across the different subsets.
However, as I hinted at in a comment above, what's important is that you ask yourself the question and indicate (in reporting the results) which steps you took to check for significance. Too many people brush these issues under the carpet or just claim significance with no further details. 
Note that some journals have banned certain forms of statistical testing because of misuse.
A: This paper by Hanley and McNeil (1982) might be helpful in finding a confidence interval between the two AUC's of the different classifiers. http://pubs.rsna.org/doi/pdf/10.1148/radiology.148.3.6878708
The paper claims that you can calculate the SE of the difference of the two AUC metrics $AUC = AUC_1 - AUC_2$ with the following formula.
$$ 
SE(AUC) = \sqrt{\frac{AUC(1-AUC)+(N_1-1)(Q_1-AUC^2)+(N_2-1)(Q_2-AUC^2)}{N_1N_2}}
$$
where
$$ 
Q_1 = \frac{AUC}{2-AUC}
$$
$$
Q_2 = \frac{2AUC^2}{1+AUC}
$$
A: You cannot determine if two measures are significantly different (in a statistical sense). Statistic significance can only be determined for sets of measures (in this case sets of measures).
This question in Cross-Validated How to statistically compare the performance of machine learning classifiers? is a good start on how to collect a set of measures and which test to use to compare classifiers. But notice that the literature posted in the accepted answer deals with accuracy not AUC. I don't know if using AUC as quality measure changes the tests. 
