Linked Questions
32 questions linked to/from What does it mean that AUC is a semi-proper scoring rule?
0
votes
0
answers
44
views
Are imbalanced groups a problem for logistic regression?
I have one group with n=13 and another with n=26 (proportion 1:2).
I have 14 features to use in the classification model.
I am using a logistic regression model.
My questions are:
Is it correct to ...
- 1
0
votes
1
answer
33
views
How likely is it that our model better than random in the upper corner of the AUC?
We're using forest-based models in a personnel selection context. For a dataset with 57 features, 230 observations, and a binary outcome, we got the following ROC curves.
This shows the first 6 folds ...
- 113
1
vote
1
answer
133
views
Relation between AUROC and threshold
As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that ...
- 521
2
votes
1
answer
137
views
Why only accuracy is used in meta and few-shot learning as evaluation parameters?
I was going through many state-of-the-art papers in Meta-learning and few-shot learning, and I found that almost all use "accuracy" as evaluation criterion. Unlike other domains like object ...
- 45
3
votes
1
answer
144
views
Is the ROC curve sufficient for rejecting the null hypotesis in binary classifications?
Problem definition
Suppose I want to test if a classifier is of any use in telling if a person is currently affected by a disease.
I have trained my classifier on a training set and now I have its ...
1
vote
0
answers
224
views
Why class-balancing techniques are sometimes useful?
There are a lot of questions here regarding when to do class balancing, or what to expect of class balancing or whether unbalanced classes are an issue at all.
Apparently the "consensus" ...
- 111
1
vote
0
answers
100
views
Why is AUC so often use to compare performance of different models in churn prediction task?
I have to build model to predict churn and when reading related work on the internet I have realized that in most of the cases the AUC is used as a metric to compare different models. That's ...
- 75
0
votes
1
answer
215
views
Choice of a loss function
Im running an xgboost model to try and find important predictors for a disease from a list of almost 1000 covariates. The prevalence of the disease in my cohort is about 10%.
Given the imbalance data, ...
- 159
1
vote
1
answer
376
views
Evaluation metric for time-series anomaly detection
My dataset is time-series sensor data and anomaly ratio is between 5% and 6%
1.
For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?
When empirically ...
0
votes
1
answer
57
views
imbalanced classes: ROC_AUC vs Precision_Recall AUC
I am dealing with a highly imbalanced classes problem. Accuracy is of course not a good performance metric in such cases, So I want to calculate either ROC AUC sore ...
- 263
3
votes
1
answer
137
views
Which metric to use to evaluate highly imbalance classification model performance
I have to do classification model to predict the possibilities of person getting cancer based on certain attributes. The data is highly imbalanced. As per client requirement I have to report model ...
- 489
2
votes
1
answer
760
views
ROC for testing goodness of fit
I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression.
I'm a bit confused by the literature where it is mostly just explained as a valid technique to ...
- 2,071
2
votes
1
answer
49
views
Performance metric for small, imbalanced, binary dataset?
I'm training an Elastic Net model on a small dataset with about 100 TRUE outcomes and 15 FALSE outcomes. I've been using ...
- 215
4
votes
1
answer
436
views
How can proper scoring rules optimize the probabilistic prediction compared to improper scoring rules?
I understand the fundamentals in the decision theory about accuracy being an improper scoring rule compared to other proper scoring rules like ...
- 448
2
votes
1
answer
1k
views
Comparing AUC and classification loss for binary outcome in LASSO cross validation
I'm analyzing biological data where I'd like to see the impact of scaled gene expression on the classification of the sample. I binarized the response variable as 0 ...
- 701