Linked Questions

2 votes
0 answers
32 views

Prove that AUROC is an improper scoring rule [duplicate]

It has been stated in many places that AUROC is an improper scoring rule.But I haven't seen anyone proving it. Does someone have a working example that shows that maximizing AUROC actually moves away ...
J.doe's user avatar
  • 359
37 votes
3 answers
45k views

Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

I have two classifiers A: naive Bayesian network B: tree (singly-connected) Bayesian network In terms of accuracy and other measures, A performs comparatively worse than B. However, when I use the R ...
Jane Wayne's user avatar
  • 1,399
40 votes
3 answers
2k views

When is it appropriate to use an improper scoring rule?

Merkle & Steyvers (2013) write: To formally define a proper scoring rule, let $f$ be a probabilistic forecast of a Bernoulli trial $d$ with true success probability $p$. Proper scoring ...
user1205901 - Слава Україні's user avatar
23 votes
3 answers
3k views

(Why) Is absolute loss not a proper scoring rule?

Brier score is a proper scoring rule and is, at least in the binary classification case, square loss. $$Brier(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N\big\vert y_i -\hat{y}_i\big\vert^2$$ Apparently this ...
Dave's user avatar
  • 63.6k
20 votes
2 answers
4k views

Is up- or down-sampling imbalanced data actually that effective? Why?

I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data. I understand that this could be useful if you're working with a binary (as opposed ...
Ben Kuhn's user avatar
  • 5,728
1 vote
2 answers
10k views

Calculate AUC using sensitivity and specificity values only

How to calculate AUC, if I have values of sensitivity and specificity for various threshold cutoffs? I have sensitivity and specificity values for 100 thresholds. ...
Dhwani Dholakia's user avatar
3 votes
2 answers
1k views

Do you need to calculate sample size to evaluate a new diagnostic test?

I am writing a grant application which will be evaluating a new diagnostic test. The test will predict whether a patient with lung fibrosis will remain stable or progress. I am using an existing ...
GhostRider's user avatar
2 votes
3 answers
1k views

Accuracy, Sensitivity, Specificity, & ROC AUC [duplicate]

In the context of predictive modeling, when comparing clasification models, What statistic should be considered more important over the others: Accuracy, sensitivity, specificity, or area under ROC ...
Nip's user avatar
  • 561
2 votes
1 answer
2k views

Comparing AUC and classification loss for binary outcome in LASSO cross validation

I'm analyzing biological data where I'd like to see the impact of scaled gene expression on the classification of the sample. I binarized the response variable as 0 ...
Atakan's user avatar
  • 761
8 votes
1 answer
478 views

Are Brier and log-loss proper or strictly proper scoring rules?

(This article nicely explains the difference between proper and proper scoring rules) According to the Wikipedia entry, and Merkle & Steyvers (2013), these are both strictly proper scoring rules. ...
Gabriel's user avatar
  • 4,342
4 votes
1 answer
522 views

How can proper scoring rules optimize the probabilistic prediction compared to improper scoring rules?

I understand the fundamentals in the decision theory about accuracy being an improper scoring rule compared to other proper scoring rules like ...
Thomas's user avatar
  • 528
2 votes
1 answer
1k views

ROC for testing goodness of fit

I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression. I'm a bit confused by the literature where it is mostly just explained as a valid technique to ...
Paze's user avatar
  • 2,311
6 votes
1 answer
615 views

Is the AUC an incoherent measure of classifier performance?

I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ...
Gabriel's user avatar
  • 4,342
2 votes
1 answer
662 views

Which evaluation metrics are mutually redundant?

Suppose we are given a confusion matrix for a binary classification: tp, fp fn, tn Now, there are lots of evaluation metrics: POD (probability of detection, aka hit rate, ...
eugen's user avatar
  • 121
0 votes
1 answer
283 views

Choice of a loss function

Im running an xgboost model to try and find important predictors for a disease from a list of almost 1000 covariates. The prevalence of the disease in my cohort is about 10%. Given the imbalance data, ...
dean's user avatar
  • 455

15 30 50 per page