32 views

### Prove that AUROC is an improper scoring rule [duplicate]

It has been stated in many places that AUROC is an improper scoring rule.But I haven't seen anyone proving it. Does someone have a working example that shows that maximizing AUROC actually moves away ...
• 357
43k views

### Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

I have two classifiers A: naive Bayesian network B: tree (singly-connected) Bayesian network In terms of accuracy and other measures, A performs comparatively worse than B. However, when I use the R ...
• 1,337
2k views

### When is it appropriate to use an improper scoring rule?

Merkle & Steyvers (2013) write: To formally define a proper scoring rule, let $f$ be a probabilistic forecast of a Bernoulli trial $d$ with true success probability $p$. Proper scoring ...
3k views

### (Why) Is absolute loss not a proper scoring rule?

Brier score is a proper scoring rule and is, at least in the binary classification case, square loss. $$Brier(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N\big\vert y_i -\hat{y}_i\big\vert^2$$ Apparently this ...
• 46.6k
3k views

### Is up- or down-sampling imbalanced data actually that effective? Why?

I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data. I understand that this could be useful if you're working with a binary (as opposed ...
• 5,598
1 vote
7k views

### Calculate AUC using sensitivity and specificity values only

How to calculate AUC, if I have values of sensitivity and specificity for various threshold cutoffs? I have sensitivity and specificity values for 100 thresholds. ...
1k views

### Do you need to calculate sample size to evaluate a new diagnostic test?

I am writing a grant application which will be evaluating a new diagnostic test. The test will predict whether a patient with lung fibrosis will remain stable or progress. I am using an existing ...
• 447
963 views

### Accuracy, Sensitivity, Specificity, & ROC AUC [duplicate]

In the context of predictive modeling, when comparing clasification models, What statistic should be considered more important over the others: Accuracy, sensitivity, specificity, or area under ROC ...
• 561
1k views

### Comparing AUC and classification loss for binary outcome in LASSO cross validation

I'm analyzing biological data where I'd like to see the impact of scaled gene expression on the classification of the sample. I binarized the response variable as 0 ...
• 671
317 views

### Are Brier and log-loss proper or strictly proper scoring rules?

(This article nicely explains the difference between proper and proper scoring rules) According to the Wikipedia entry, and Merkle & Steyvers (2013), these are both strictly proper scoring rules. ...
• 3,418
396 views

### How can proper scoring rules optimize the probabilistic prediction compared to improper scoring rules?

I understand the fundamentals in the decision theory about accuracy being an improper scoring rule compared to other proper scoring rules like ...
• 418
684 views

### ROC for testing goodness of fit

I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression. I'm a bit confused by the literature where it is mostly just explained as a valid technique to ...
• 2,071
386 views

### Which evaluation metrics are mutually redundant?

Suppose we are given a confusion matrix for a binary classification: tp, fp fn, tn Now, there are lots of evaluation metrics: POD (probability of detection, aka hit rate, ...
• 121
191 views

### Choice of a loss function

Im running an xgboost model to try and find important predictors for a disease from a list of almost 1000 covariates. The prevalence of the disease in my cohort is about 10%. Given the imbalance data, ...
• 127
1 vote
258 views

### How to assess a model where you are interested in the probability output

I know that we assess performance of classifiers typically with metrics like accuracy, ROC, etc. typically because we want to know whether or not a classifier can accurately predict an outcome. But, ...
• 11
383 views

### Is the AUC an incoherent measure of classifier performance?

I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ...
• 3,418
1 vote
352 views

### Evaluation metric for time-series anomaly detection

My dataset is time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ? When empirically ...
1 vote
138 views

### Can't .632+ rule be computed for any kind of outcome and prediction score?

It seems that the R packages I found around for computing the .632+ estimation of prediction error work only with categorical outcomes. Why is that? Looking at the formulas in Efron 1997 paper it ...
• 2,433
1 vote
215 views

### Why class-balancing techniques are sometimes useful?

There are a lot of questions here regarding when to do class balancing, or what to expect of class balancing or whether unbalanced classes are an issue at all. Apparently the "consensus" ...
• 111
122 views

### Which metric to use to evaluate highly imbalance classification model performance

I have to do classification model to predict the possibilities of person getting cancer based on certain attributes. The data is highly imbalanced. As per client requirement I have to report model ...
• 489
1 vote
115 views

### Relation between AUROC and threshold

As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that ...
• 521
111 views

### Why only accuracy is used in meta and few-shot learning as evaluation parameters?

I was going through many state-of-the-art papers in Meta-learning and few-shot learning, and I found that almost all use "accuracy" as evaluation criterion. Unlike other domains like object ...
117 views

### Is the ROC curve sufficient for rejecting the null hypotesis in binary classifications?

Problem definition Suppose I want to test if a classifier is of any use in telling if a person is currently affected by a disease. I have trained my classifier on a training set and now I have its ...
103 views

### evaluating logistic regression's performance

I am working on the scoring model and I aim to predict the probability of default. I have, say m, different candidate Logistic Regression models $M_{1}, \dots, M_{m}$ and I would like to choose the ...
• 63
55 views

### Performance evaluation

I'd like to test the performance of a penalized regression. I did three separate regressions for each response variable (one numerical, one binomial and one multinomial). I was checking this link, and ...
1 vote
91 views

### Why is AUC so often use to compare performance of different models in churn prediction task?

I have to build model to predict churn and when reading related work on the internet I have realized that in most of the cases the AUC is used as a metric to compare different models. That's ...
• 75
48 views

### Performance metric for small, imbalanced, binary dataset?

I'm training an Elastic Net model on a small dataset with about 100 TRUE outcomes and 15 FALSE outcomes. I've been using ...
57 views

### imbalanced classes: ROC_AUC vs Precision_Recall AUC

I am dealing with a highly imbalanced classes problem. Accuracy is of course not a good performance metric in such cases, So I want to calculate either ROC AUC sore ...
• 253
31 views

### How likely is it that our model better than random in the upper corner of the AUC?

We're using forest-based models in a personnel selection context. For a dataset with 57 features, 230 observations, and a binary outcome, we got the following ROC curves. This shows the first 6 folds ...
• 113