30 questions linked to/from What does it mean that AUC is a semi-proper scoring rule?
Are imbalanced groups a problem for logistic regression?
I have one group with n=13 and another with n=26 (proportion 1:2). I have 14 features to use in the classification model. I am using a logistic regression model. My questions are: Is it correct to ...
How likely is it that our model better than random in the upper corner of the AUC?
We're using forest-based models in a personnel selection context. For a dataset with 57 features, 230 observations, and a binary outcome, we got the following ROC curves. This shows the first 6 folds ...
Relation between AUROC and threshold
As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that ...
Why only accuracy is used in meta and few-shot learning as evaluation parameters?
I was going through many state-of-the-art papers in Meta-learning and few-shot learning, and I found that almost all use "accuracy" as evaluation criterion. Unlike other domains like object ...
Is the ROC curve sufficient for rejecting the null hypotesis in binary classifications?
Problem definition Suppose I want to test if a classifier is of any use in telling if a person is currently affected by a disease. I have trained my classifier on a training set and now I have its ...
Why class-balancing techniques are sometimes useful?
There are a lot of questions here regarding when to do class balancing, or what to expect of class balancing or whether unbalanced classes are an issue at all. Apparently the "consensus" ...
Why is AUC so often use to compare performance of different models in churn prediction task?
I have to build model to predict churn and when reading related work on the internet I have realized that in most of the cases the AUC is used as a metric to compare different models. That's ...
Choice of a loss function
Im running an xgboost model to try and find important predictors for a disease from a list of almost 1000 covariates. The prevalence of the disease in my cohort is about 10%. Given the imbalance data, ...
Evaluation metric for time-series anomaly detection
My dataset is time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ? When empirically ...
imbalanced classes: ROC_AUC vs Precision_Recall AUC
I am dealing with a highly imbalanced classes problem. Accuracy is of course not a good performance metric in such cases, So I want to calculate either ROC AUC sore ...
Which metric to use to evaluate highly imbalance classification model performance
I have to do classification model to predict the possibilities of person getting cancer based on certain attributes. The data is highly imbalanced. As per client requirement I have to report model ...
ROC for testing goodness of fit
I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression. I'm a bit confused by the literature where it is mostly just explained as a valid technique to ...
Performance metric for small, imbalanced, binary dataset?
I'm training an Elastic Net model on a small dataset with about 100 TRUE outcomes and 15 FALSE outcomes. I've been using ...
How can proper scoring rules optimize the probabilistic prediction compared to improper scoring rules?
I understand the fundamentals in the decision theory about accuracy being an improper scoring rule compared to other proper scoring rules like ...
Comparing AUC and classification loss for binary outcome in LASSO cross validation
I'm analyzing biological data where I'd like to see the impact of scaled gene expression on the classification of the sample. I binarized the response variable as 0 ...