0
$\begingroup$

I have a classifier for a binary problem. That has outputs between 0 and 1 for predictions for the two class A or B (for example sunny, not sunny). The classifier has ran on 5 unique folds of the data set and in each fold I have the outcome of the classifier as a value between 0 and 1 for each class, Class A (for example 0.83) and Class B (0.17). I also know the true label for that classification attempt.

For each fold I also have Cohen's Kappa, Weighted Cross Entropy, confusion matrix, Precision, Recall and F1.

Is there anyway to calculate the AUC or approximate it using the data I have at hand?

Many thanks, Mo

$\endgroup$
2
  • 2
    $\begingroup$ If the output is continuous, you are not using a classifier but rather something like a probability model. And none of the measures you listed are proper accuracy scoring rules. See this and this. And unless your sample size is huge you may need to do 100 repeats of 10-fold cross-validation to get adequate precision for model performance metrics. $\endgroup$ – Frank Harrell Sep 30 '19 at 11:16
  • $\begingroup$ Thanks for that Frank. $\endgroup$ – Mohammad Sep 30 '19 at 11:28
1
$\begingroup$

You can use the estimator

$$\widehat{AUC}_k = \frac{\sum_{i:y_i = B}\sum_{j:y_j = A}\mathcal{I}(\hat p^A_i < \hat p^A_j)}{\sum_{i:y_i = B}\sum_{j:y_j = A} 1},$$ where $\hat p^A_i$ is the predicted probability of observation $i$ belonging to class A, computed for each of the $5$ folds, and then average these to get an estimate of the out of sample AUC as

$$\widehat{AUC} = \frac{1}{5}\sum_{k=1}^5\widehat{AUC}_k.$$

$\endgroup$
2
  • $\begingroup$ Hi thank you for your reply, can I kindly ask what is "I" and what is the denominator of that fraction? $\endgroup$ – Mohammad Sep 30 '19 at 11:00
  • $\begingroup$ $\mathcal{I}$ is an indicator function, i.e., a function that is equal to 1 whenever $\hat p_i^A < \hat p_j^A$, and 0 otherwise. The denominator is the sum over all the observations of both classes over the constant $1$. Which will be the number of observations that belong to class A times the number of observations that belong to class B. $\endgroup$ – Simon Boge Brant Sep 30 '19 at 11:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.