0
$\begingroup$

I have a classifier for a binary problem. That has outputs between 0 and 1 for predictions for the two class A or B (for example sunny, not sunny). The classifier has ran on 5 unique folds of the data set and in each fold I have the outcome of the classifier as a value between 0 and 1 for each class, Class A (for example 0.83) and Class B (0.17). I also know the true label for that classification attempt.

For each fold I also have Cohen's Kappa, Weighted Cross Entropy, confusion matrix, Precision, Recall and F1.

Is there anyway to calculate the AUC or approximate it using the data I have at hand?

Many thanks, Mo

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ If the output is continuous, you are not using a classifier but rather something like a probability model. And none of the measures you listed are proper accuracy scoring rules. See this and this. And unless your sample size is huge you may need to do 100 repeats of 10-fold cross-validation to get adequate precision for model performance metrics. $\endgroup$
    – Frank Harrell
    Sep 30, 2019 at 11:16
  • $\begingroup$ Thanks for that Frank. $\endgroup$
    – Mohammad
    Sep 30, 2019 at 11:28

1 Answer 1

Reset to default
1
$\begingroup$

You can use the estimator

$$\widehat{AUC}_k = \frac{\sum_{i:y_i = B}\sum_{j:y_j = A}\mathcal{I}(\hat p^A_i < \hat p^A_j)}{\sum_{i:y_i = B}\sum_{j:y_j = A} 1},$$ where $\hat p^A_i$ is the predicted probability of observation $i$ belonging to class A, computed for each of the $5$ folds, and then average these to get an estimate of the out of sample AUC as

$$\widehat{AUC} = \frac{1}{5}\sum_{k=1}^5\widehat{AUC}_k.$$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi thank you for your reply, can I kindly ask what is "I" and what is the denominator of that fraction? $\endgroup$
    – Mohammad
    Sep 30, 2019 at 11:00
  • $\begingroup$ $\mathcal{I}$ is an indicator function, i.e., a function that is equal to 1 whenever $\hat p_i^A < \hat p_j^A$, and 0 otherwise. The denominator is the sum over all the observations of both classes over the constant $1$. Which will be the number of observations that belong to class A times the number of observations that belong to class B. $\endgroup$
    – Simon Boge Brant
    Sep 30, 2019 at 11:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.