Linked Questions
30 questions linked to/from What does it mean that AUC is a semi-proper scoring rule?
1
vote
1
answer
352
views
Evaluation metric for time-series anomaly detection
My dataset is time-series sensor data and anomaly ratio is between 5% and 6%
1.
For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?
When empirically ...
CommunityBot
- 1
0
votes
0
answers
35
views
Are imbalanced groups a problem for logistic regression?
I have one group with n=13 and another with n=26 (proportion 1:2).
I have 14 features to use in the classification model.
I am using a logistic regression model.
My questions are:
Is it correct to ...
- 1
18
votes
2
answers
3k
views
Is up- or down-sampling imbalanced data actually that effective? Why?
I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data.
I understand that this could be useful if you're working with a binary (as opposed ...
- 5,404
20
votes
3
answers
3k
views
(Why) Is absolute loss not a proper scoring rule?
Brier score is a proper scoring rule and is, at least in the binary classification case, square loss.
$$Brier(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N\big\vert y_i -\hat{y}_i\big\vert^2$$
Apparently this ...
- 7,621
0
votes
1
answer
31
views
How likely is it that our model better than random in the upper corner of the AUC?
We're using forest-based models in a personnel selection context. For a dataset with 57 features, 230 observations, and a binary outcome, we got the following ROC curves.
This shows the first 6 folds ...
- 77.5k
1
vote
1
answer
115
views
Relation between AUROC and threshold
As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that ...
- 109k
2
votes
1
answer
111
views
Why only accuracy is used in meta and few-shot learning as evaluation parameters?
I was going through many state-of-the-art papers in Meta-learning and few-shot learning, and I found that almost all use "accuracy" as evaluation criterion. Unlike other domains like object ...
- 34.7k
1
vote
0
answers
215
views
Why class-balancing techniques are sometimes useful?
There are a lot of questions here regarding when to do class balancing, or what to expect of class balancing or whether unbalanced classes are an issue at all.
Apparently the "consensus" ...
- 1,063
3
votes
0
answers
117
views
Is the ROC curve sufficient for rejecting the null hypotesis in binary classifications?
Problem definition
Suppose I want to test if a classifier is of any use in telling if a person is currently affected by a disease.
I have trained my classifier on a training set and now I have its ...
- 71k
1
vote
0
answers
91
views
Why is AUC so often use to compare performance of different models in churn prediction task?
I have to build model to predict churn and when reading related work on the internet I have realized that in most of the cases the AUC is used as a metric to compare different models. That's ...
- 75
0
votes
1
answer
192
views
Choice of a loss function
Im running an xgboost model to try and find important predictors for a disease from a list of almost 1000 covariates. The prevalence of the disease in my cohort is about 10%.
Given the imbalance data, ...
- 109k
37
votes
3
answers
2k
views
When is it appropriate to use an improper scoring rule?
Merkle & Steyvers (2013) write:
To formally define a proper scoring rule, let $f$ be a probabilistic
forecast of a Bernoulli trial $d$ with true success probability $p$.
Proper scoring ...
- 336
0
votes
1
answer
57
views
imbalanced classes: ROC_AUC vs Precision_Recall AUC
I am dealing with a highly imbalanced classes problem. Accuracy is of course not a good performance metric in such cases, So I want to calculate either ROC AUC sore ...
- 109k
3
votes
1
answer
122
views
Which metric to use to evaluate highly imbalance classification model performance
I have to do classification model to predict the possibilities of person getting cancer based on certain attributes. The data is highly imbalanced. As per client requirement I have to report model ...
- 1,775
6
votes
1
answer
317
views
Are Brier and log-loss proper or strictly proper scoring rules?
(This article nicely explains the difference between proper and proper scoring rules)
According to the Wikipedia entry, and Merkle & Steyvers (2013), these are both strictly proper scoring rules. ...
- 109k