Linked Questions
30 questions linked to/from What does it mean that AUC is a semi-proper scoring rule?
37
votes
3
answers
2k
views
When is it appropriate to use an improper scoring rule?
Merkle & Steyvers (2013) write:
To formally define a proper scoring rule, let $f$ be a probabilistic
forecast of a Bernoulli trial $d$ with true success probability $p$.
Proper scoring ...
36
votes
3
answers
43k
views
Why is AUC higher for a classifier that is less accurate than for one that is more accurate?
I have two classifiers
A: naive Bayesian network
B: tree (singly-connected) Bayesian network
In terms of accuracy and other measures, A performs comparatively worse than B. However, when I use the R ...
- 1,337
20
votes
3
answers
3k
views
(Why) Is absolute loss not a proper scoring rule?
Brier score is a proper scoring rule and is, at least in the binary classification case, square loss.
$$Brier(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N\big\vert y_i -\hat{y}_i\big\vert^2$$
Apparently this ...
- 46.6k
18
votes
2
answers
3k
views
Is up- or down-sampling imbalanced data actually that effective? Why?
I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data.
I understand that this could be useful if you're working with a binary (as opposed ...
- 5,598
6
votes
1
answer
317
views
Are Brier and log-loss proper or strictly proper scoring rules?
(This article nicely explains the difference between proper and proper scoring rules)
According to the Wikipedia entry, and Merkle & Steyvers (2013), these are both strictly proper scoring rules. ...
- 3,418
4
votes
1
answer
397
views
How can proper scoring rules optimize the probabilistic prediction compared to improper scoring rules?
I understand the fundamentals in the decision theory about accuracy being an improper scoring rule compared to other proper scoring rules like ...
- 418
3
votes
2
answers
1k
views
Do you need to calculate sample size to evaluate a new diagnostic test?
I am writing a grant application which will be evaluating a new diagnostic test. The test will predict whether a patient with lung fibrosis will remain stable or progress. I am using an existing ...
- 447
3
votes
1
answer
122
views
Which metric to use to evaluate highly imbalance classification model performance
I have to do classification model to predict the possibilities of person getting cancer based on certain attributes. The data is highly imbalanced. As per client requirement I have to report model ...
- 489
3
votes
0
answers
117
views
Is the ROC curve sufficient for rejecting the null hypotesis in binary classifications?
Problem definition
Suppose I want to test if a classifier is of any use in telling if a person is currently affected by a disease.
I have trained my classifier on a training set and now I have its ...
3
votes
0
answers
384
views
Is the AUC an incoherent measure of classifier performance?
I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ...
- 3,418
2
votes
3
answers
966
views
Accuracy, Sensitivity, Specificity, & ROC AUC [duplicate]
In the context of predictive modeling, when comparing clasification models, What statistic should be considered more important over the others: Accuracy, sensitivity, specificity, or area under ROC ...
- 561
2
votes
1
answer
687
views
ROC for testing goodness of fit
I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression.
I'm a bit confused by the literature where it is mostly just explained as a valid technique to ...
- 2,071
2
votes
1
answer
1k
views
Comparing AUC and classification loss for binary outcome in LASSO cross validation
I'm analyzing biological data where I'd like to see the impact of scaled gene expression on the classification of the sample. I binarized the response variable as 0 ...
- 671
2
votes
1
answer
394
views
Which evaluation metrics are mutually redundant?
Suppose we are given a confusion matrix for a binary classification:
tp, fp
fn, tn
Now, there are lots of evaluation metrics:
POD (probability of detection, aka hit rate, ...
- 121
2
votes
1
answer
113
views
Why only accuracy is used in meta and few-shot learning as evaluation parameters?
I was going through many state-of-the-art papers in Meta-learning and few-shot learning, and I found that almost all use "accuracy" as evaluation criterion. Unlike other domains like object ...
- 45