Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity I have a test set of 100 cases and two classifiers.
I generated predictions and computed ROC AUC, sensitivity and specificity for both classifiers.
Question 1: How can I compute p-value to check if one is significantly better than the other with respect to all scores (ROC AUC, sensitivity, specificity)?

Now, for the same test set of 100 cases, I have different and independent feature assignments for each case.
This is because my features are fixed but subjective and provided by multiple (5) subjects.
So, I evaluated my two classifiers again for 5 "versions" of my test set and obtained 5 ROC AUCs, 5 sensitivities and 5 specificities for both classifiers.
Then, I computed the mean of each performance measure for 5 subjects (mean ROC AUC, mean sensitivity and mean specificity) for both classifiers.
Question 2: How can I compute p-value to check if one is significantly better than the other with respect to mean scores (mean ROC AUC, mean sensitivity, mean specificity)?

Answers with some example python (preferably) or MatLab code are more than welcome.
 A: Let me keep the answer short, because this guide does explain a lot more and better.
Basically, you have your number of True Postives ($nTP$) and number of True Negatives ($nTN$). Also you have your AUC, A. The standard error of this A is:
$\texttt{SE}_A = \sqrt{\frac{A(1-A) + (nTP-1)(Q_1 - A^2)+(nTN-1)(Q_2 - A^2)}{nTP \cdot nTN}}$
with $Q_1 =  A / (2 - A)$ and $Q_2 = 2A^2 / (1 + A)$.
To compare two AUCs you need to compute the SE of them both using:
$\texttt{SE}_{A_1 - A_2} = \sqrt{(SE_{A_1})^2 + (SE_{A_2})^2 - 2r\cdot (SE_{A_1})(SE_{A_2})}$
where $r$ is a quantity that represents the correlation induced between the two areas by the study of the same set of cases. If your cases are different, then $r=0$;  otherwise you need to look it up (Table 1, page 3 in freely available article).
Given that you compute the $z$-Score by
$z = (A_1 - A_2) / SE_{A_1-A_2}$
From there you can compute p-value using probability density of a standard normal distribution. Or simply use this calculator.
This hopefully answers Question 1. - at least the part comparing AUCs. Sens/Spec is already covered by the ROC/AUC in some way. Otherwise, the answer I think lies in the Question 2.
As for Question 2, Central Limit Theorem tells us that your summary statistic would follow a normal distribution. Hence, I would think a simple t-test would suffice (5 measures of one classifier against 5 measures of the second classifier where measures could be AUC, sens, spec)
Edit: corrected formula for $\texttt{SE}$ ($\ldots - 2r \ldots$)
A: For Question 1, @Sycorax provided a comprehensive answer.
For Question 2, to the best of my knowledge, averaging predictions from subjects is incorrect.
I decided to use bootstrapping to compute p-values and compare models.
In this case, the procedure is as follows:
For N iterations:
  sample 5 subjects with replacement
  sample 100 test cases with replacement
  compute mean performance of sampled subjects on sampled cases for model M1
  compute mean performance of sampled subjects on sampled cases for model M2
  take the difference of mean performance between M1 and M2
p-value equals to the proportion of differences smaller or equal than 0

This procedure performs one-tailed test and assumes that M1 mean performance > M2 mean performance.
A Python implementation of bootstrapping for computing p-values comparing multiple readers can be found in this GitHub repo: https://github.com/mateuszbuda/ml-stat-util
