Linked Questions
32 questions linked to/from What does it mean that AUC is a semi-proper scoring rule?
0
votes
0
answers
44
views
Are imbalanced groups a problem for logistic regression?
I have one group with n=13 and another with n=26 (proportion 1:2).
I have 14 features to use in the classification model.
I am using a logistic regression model.
My questions are:
Is it correct to ...
- 1
0
votes
1
answer
33
views
How likely is it that our model better than random in the upper corner of the AUC?
We're using forest-based models in a personnel selection context. For a dataset with 57 features, 230 observations, and a binary outcome, we got the following ROC curves.
This shows the first 6 folds ...
- 113
1
vote
1
answer
133
views
Relation between AUROC and threshold
As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that ...
- 521
2
votes
1
answer
138
views
Why only accuracy is used in meta and few-shot learning as evaluation parameters?
I was going through many state-of-the-art papers in Meta-learning and few-shot learning, and I found that almost all use "accuracy" as evaluation criterion. Unlike other domains like object ...
- 45
3
votes
1
answer
144
views
Is the ROC curve sufficient for rejecting the null hypotesis in binary classifications?
Problem definition
Suppose I want to test if a classifier is of any use in telling if a person is currently affected by a disease.
I have trained my classifier on a training set and now I have its ...
1
vote
0
answers
224
views
Why class-balancing techniques are sometimes useful?
There are a lot of questions here regarding when to do class balancing, or what to expect of class balancing or whether unbalanced classes are an issue at all.
Apparently the "consensus" ...
- 111
1
vote
0
answers
100
views
Why is AUC so often use to compare performance of different models in churn prediction task?
I have to build model to predict churn and when reading related work on the internet I have realized that in most of the cases the AUC is used as a metric to compare different models. That's ...
- 75
0
votes
1
answer
216
views
Choice of a loss function
Im running an xgboost model to try and find important predictors for a disease from a list of almost 1000 covariates. The prevalence of the disease in my cohort is about 10%.
Given the imbalance data, ...
- 159
1
vote
1
answer
376
views
Evaluation metric for time-series anomaly detection
My dataset is time-series sensor data and anomaly ratio is between 5% and 6%
1.
For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?
When empirically ...
0
votes
1
answer
57
views
imbalanced classes: ROC_AUC vs Precision_Recall AUC
I am dealing with a highly imbalanced classes problem. Accuracy is of course not a good performance metric in such cases, So I want to calculate either ROC AUC sore ...
- 263
3
votes
1
answer
137
views
Which metric to use to evaluate highly imbalance classification model performance
I have to do classification model to predict the possibilities of person getting cancer based on certain attributes. The data is highly imbalanced. As per client requirement I have to report model ...
- 489
2
votes
1
answer
762
views
ROC for testing goodness of fit
I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression.
I'm a bit confused by the literature where it is mostly just explained as a valid technique to ...
- 2,071
2
votes
1
answer
49
views
Performance metric for small, imbalanced, binary dataset?
I'm training an Elastic Net model on a small dataset with about 100 TRUE outcomes and 15 FALSE outcomes. I've been using ...
- 215
4
votes
1
answer
436
views
How can proper scoring rules optimize the probabilistic prediction compared to improper scoring rules?
I understand the fundamentals in the decision theory about accuracy being an improper scoring rule compared to other proper scoring rules like ...
- 448
2
votes
1
answer
1k
views
Comparing AUC and classification loss for binary outcome in LASSO cross validation
I'm analyzing biological data where I'd like to see the impact of scaled gene expression on the classification of the sample. I binarized the response variable as 0 ...
- 701
6
votes
1
answer
355
views
Are Brier and log-loss proper or strictly proper scoring rules?
(This article nicely explains the difference between proper and proper scoring rules)
According to the Wikipedia entry, and Merkle & Steyvers (2013), these are both strictly proper scoring rules. ...
- 3,440
20
votes
3
answers
3k
views
(Why) Is absolute loss not a proper scoring rule?
Brier score is a proper scoring rule and is, at least in the binary classification case, square loss.
$$Brier(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N\big\vert y_i -\hat{y}_i\big\vert^2$$
Apparently this ...
- 51.1k
3
votes
1
answer
439
views
Is the AUC an incoherent measure of classifier performance?
I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ...
- 3,440
1
vote
1
answer
41
views
Comparison of ML models
There are two models f, g trained on some labelled (x,y), where y has 2 classes.
During testing they correctly predict the same unseen samples. However, the probability they output are different. So ...
1
vote
1
answer
146
views
Can't .632+ rule be computed for any kind of outcome and prediction score?
It seems that the R packages I found around for computing the .632+ estimation of prediction error work only with categorical outcomes.
Why is that? Looking at the formulas in Efron 1997 paper it ...
- 2,513
2
votes
1
answer
451
views
Which evaluation metrics are mutually redundant?
Suppose we are given a confusion matrix for a binary classification:
tp, fp
fn, tn
Now, there are lots of evaluation metrics:
POD (probability of detection, aka hit rate, ...
- 121
2
votes
3
answers
1k
views
Accuracy, Sensitivity, Specificity, & ROC AUC [duplicate]
In the context of predictive modeling, when comparing clasification models, What statistic should be considered more important over the others: Accuracy, sensitivity, specificity, or area under ROC ...
- 561
2
votes
0
answers
32
views
Prove that AUROC is an improper scoring rule [duplicate]
It has been stated in many places that AUROC is an improper scoring rule.But I haven't seen anyone proving it. Does someone have a working example that shows that maximizing AUROC actually moves away ...
- 357
1
vote
2
answers
8k
views
Calculate AUC using sensitivity and specificity values only
How to calculate AUC, if I have values of sensitivity and specificity for various threshold cutoffs?
I have sensitivity and specificity values for 100 thresholds.
...
1
vote
1
answer
117
views
evaluating logistic regression's performance
I am working on the scoring model and I aim to predict the probability of default. I have, say m, different candidate Logistic Regression models $M_{1}, \dots, M_{m}$ and I would like to choose the ...
- 73
3
votes
2
answers
1k
views
Do you need to calculate sample size to evaluate a new diagnostic test?
I am writing a grant application which will be evaluating a new diagnostic test. The test will predict whether a patient with lung fibrosis will remain stable or progress. I am using an existing ...
- 447
0
votes
2
answers
55
views
Performance evaluation
I'd like to test the performance of a penalized regression. I did three separate regressions for each response variable (one numerical, one binomial and one multinomial). I was checking this link, and ...
- 171
1
vote
1
answer
263
views
How to assess a model where you are interested in the probability output
I know that we assess performance of classifiers typically with metrics like accuracy, ROC, etc. typically because we want to know whether or not a classifier can accurately predict an outcome. But, ...
- 11
4
votes
1
answer
362
views
Relation between L2 loss and accuracy
Given a classification problem with k classes, suppose a model outputting a probability distribution over the classes that uses some gradient-based learning method is used (yes, in my case it's neural ...
- 245
38
votes
3
answers
2k
views
When is it appropriate to use an improper scoring rule?
Merkle & Steyvers (2013) write:
To formally define a proper scoring rule, let $f$ be a probabilistic
forecast of a Bernoulli trial $d$ with true success probability $p$.
Proper scoring ...
