Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
542
questions
0
votes
0
answers
11
views
Why PR score is down when balanced accuracy is good?
I just read this discussion here and here.
I have a dataset of 977 records where class proportion is 77:23.
My balanced accuracy is 75.5, ...
- 1,758
2
votes
2
answers
38
views
How is AUC helpful when we only need one threshold of a classifier
AUC is a summation of performance at different thresholds, but do we only care about a good performance at one threshold?
Imagine a classifier with a low ROC but shots up at point of a low FP and high ...
- 21
0
votes
0
answers
15
views
Understand AUCs of different models
I'm testing two models against each other:
provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %.
provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
- 2,187
3
votes
1
answer
156
views
If the AUC score is 100 percent can F1 value be 99.94 percent?
If the AUC score is 100 percent can the F1 value be 99.94 percent? I would expect 100 percent, too.
- 189
0
votes
0
answers
9
views
Why R survAUC::AUC functions require both training and test data?
I am familiar with binary classification models, where the AUC is the metric from the area under the specificity-sensitivity curve, it indicates the performance on a dataset.
Now, I'd like to assess ...
- 21
0
votes
0
answers
23
views
Make ROC graph with my problematic [duplicate]
For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic.
I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
- 1
0
votes
0
answers
13
views
Deriving Bayesian Credible Intervals for AUC using R brms
I am trying to estimate the posterior distribution for the AUC of a predictive biomarker using R brms. However, whenever I calculate the AUC using the posterior distribution of the model parameters, ...
- 43
0
votes
1
answer
23
views
How to compare multiple survival scores regarding its accuracy?
I have a clinical dataset which has already common prognosis scores, like characteristic tumor markers, tumor classification, etc. However I developed a new score which is able to predict patients' ...
0
votes
1
answer
16
views
How to interpret this confusion matrix and roc curve?
I got these two images for evaluating a RF:
I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %?
By looking only at ...
- 2,187
0
votes
0
answers
14
views
AUC first drops before starting to grow
I have a very imbalanced classification problem, 99% vs 1%, and I am using logistic regression in pytorch. I often see the initial weights, randomly initialized, achieve AUC of 60-70% before any ...
- 11
1
vote
0
answers
22
views
Can I use the AUR under the ROC on unbalanced test data?
I have split my data into training and test data, built several prediction models and now I want to evaluate the models using the test data set.The data is very unbalanced so I balanced the training ...
- 11
2
votes
1
answer
49
views
AUC - Logistic Regression versus LDA, and Naive Bayes
everyone!
I am a newbie on machine learning, and I am now interested on classification modeling.
I used logistic regression, linear discriminant analysis (LDA), and naive Bayes on my notebook DataCamp ...
2
votes
1
answer
121
views
Appropriate way to get cross validated performance metrics
For cross-validation of a logistic regression classifier, it seems to me that there are a number of different approaches to calculating each performance metric:
The performance metric is calculated ...
- 144
3
votes
2
answers
59
views
Difference between ROC-AUC and Multiclass AUC (MAUC)
I am trying to understand the interpretation of these metrics in a multiclass scenario: ROC-AUC and MAUC. Scikit-learn provides ...
- 231
0
votes
0
answers
66
views
Comparing the difference in test AUC between different machine learning models using Delong's test
I'm in the process of developing four machine learning models (Neural network, logistic regression, XGBoost, SVM) on the same training data using the same exact features (10 features in my model) and ...
1
vote
1
answer
52
views
AUC between estimated lines
Let's say you have a lmer model that test the drug effect of a set of rats with a set of rats (Control):
lme1 <- lmer(lVolume ~ Days*Drug + (Days|Drug))
where ...
- 111
0
votes
0
answers
23
views
Can anyone explain how to calculate AUC_DS, AUC_BW, and AUC_0 mentioned in this image?
I know AUC is for binary classification, but in a paper, the authors seemingly used AUC_DS, AUC_BW, and AUC_0 to compare vector values, like
ground truth of one sample [3,2,1,0,1,2,3]
prediction value ...
- 1
0
votes
0
answers
26
views
My data can be approximated with normal distribution (multimodal). How can I find the reasons and explain this behaviour?
I use DeLonge method to compare two ROC AUCS. The result of it is Z-score.
Both ROC AUCs obtained from LDA (linear discriminant analysis) from sklearn package. The ...
- 1
0
votes
0
answers
69
views
Precision-Recall curve and AUC in multi-class problems using R
I'm trying to evaluate a RandomForest model for multi-class classification using the Area Under Precision-Recall Curve. I need to plot the PR curve of each class and the micro and macro average and ...
- 113
0
votes
1
answer
66
views
How to report AUC from Cox Regression?
I need to calculate the AUC for a Cox regression model. SAS is giving me a time dependent AUC as below. How do I report this for the paper?
Thanks!!
- 55
0
votes
0
answers
23
views
Feature Selection + Unpenalized Logistic Regression AUC
I am building a predictive model using a highly-dimensional dataset. To reduce dimensionality and reduce overfitting, I used LASSO regression to select features.
...
- 1
0
votes
1
answer
20
views
Ridge regression coefficients show model importance but the model evaulation not
I have performed two ridge logistic regressions in R to check which of the two models perform better. From the first look of the coefficients, it looks like model1 ...
- 332
2
votes
1
answer
78
views
If going with the opposite prediction of a bad predictor gives good predictions, why not do that?
Let’s restrict our consideration to binary outcomes.
I have a friend who is terrible at predicting the future, always predicting the opposite of what winds up happening. For instance, my friend ...
- 31k
0
votes
0
answers
31
views
How to compute ROC AUC for a method that uses two models?
To compare my method with others I'm trying to compute its AUC but I got a bit confused on how to do this for my case. My method uses a model that classifies an image as class A or B, after that if ...
1
vote
2
answers
74
views
Proper way to finding area under ROC curve (AUC)
I have data of 200 persons with 2 variables:
1. C_level: Blood level of a chemical C (a numeric value)
2. Disease: Yes or No
I want to know if C_level can be used ...
- 8,993
4
votes
4
answers
553
views
Validity of AUC for binary categorical variables
Scikit-learn function roc_auc_score can be used to get area under curve (AUC) of ROC curve. This score is generally used for numeric predictors' value in predicting ...
- 8,993
2
votes
1
answer
40
views
Is it valid to do ROC analysis without using test data?
I am trying to do an ROC analysis to check whether a particular biomarker has any diagnostic power for screening a condition. Since I have only small dataset of 100. I am not splitting the data into ...
- 811
0
votes
0
answers
17
views
Can you check the significance of your AUC values? [duplicate]
I have computed some AUC values from the ROC curve based on logistic regressions. Firstly, I have divided my two datasets (D1, D2...
- 332
0
votes
0
answers
30
views
ROC AUC for average and geometric mean of two models
Suppose I have two binary classification models built on the same dataset.
One has ROC AUC of 0.86, another one has ROC AUC of 0.92.
What will be the ROC AUC for the average of two models and their ...
- 1
0
votes
1
answer
81
views
How to simulate a calibrated prediction model given prevalence and auc
I want to be able to simulate a prediction model given some prevalence of the event and the AUC of the model. I followed the method proposed here but, although this works for giving AUC and predicted ...
- 33
0
votes
1
answer
17
views
AUC Instability when averaging over individual readers
I am trying to evaluated an algorithm that supports doctors when making a diagnosis.
I have recruited 10 doctors.
I have 50 training examples.
Each doctor is randomly assigned 25 cases to review alone ...
- 1
0
votes
0
answers
16
views
Does the imbalance in my data affect model quality in this case?
I have two datasets, used for classification task: training and validation one.
Both of them are equally imbalanced, with about 24% of target value equal 1 and 75% of target value equal 0.
I am using ...
- 113
0
votes
0
answers
41
views
AUC and misclassification rate [duplicate]
why do you think that is possible? I compare decision tree with neural network in SAS EM and it chooses DT as the best model, based on misclassification rate, I guess. But AUC is higher for neural ...
- 3
1
vote
0
answers
69
views
PRAUC curve Interpretation on imbalanced data
My training data is undersampled to a positive to negative ratio of 5%. I observe a PRAUC of .4 on my training data. When I test the model on real-world data where the positive to negative ratio is .5%...
- 21
2
votes
1
answer
20
views
Model selection in presence of overfitting - better test or closer train
Suppose I have a tree-based model (Random Forest for the sake of the example) and I play with a regularization parameter (tree depth) to fight overfitting.
Eventually I can come up with two models - ...
- 318
1
vote
2
answers
143
views
Training AUC below 0.5
I've trained a logistic regression using a small number of predictors - pseudo R-squared is only 0.1 but I have significant terms and a nice low p value for the model. However, even on its own ...
- 193
3
votes
1
answer
37
views
The different usage of ROC between diagnotic tests and machine learning
I am currently very confused about the ROC usage in diagnostic tests and machine learning.
In the scenario of medical diagnostic studies, many tutorial does not mention the data split procedure as ...
- 43
3
votes
1
answer
41
views
Derivation of closed-form ROC expression under binormal assumptions
It's a known result that, under binormality assumptions, the area under the ROC curve (AUC) for a binary classifier has the following closed form. Formally, define the class conditional mean and ...
- 1,066
0
votes
0
answers
8
views
Is it possible to build an imputed 2-variable AUC model
I am building a 2-variable predictive model but there are some missing data for both of my predictors. Is it possible to combine MICE code with AUC code to get something called "imputed AUC model&...
- 311
1
vote
0
answers
29
views
How to choose the balance between model fit vs AUC (diagnostic accuracy)?
I would like to know how we can choose between model fit (calibration) vs AUC when building the predictive model. For example, if I have one predictor which improves the model fit but results in a ...
- 311
1
vote
1
answer
47
views
Relation between AUROC and threshold
As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that ...
- 481
0
votes
2
answers
197
views
AUC with different threshold
I know AUC is supposed to be independent on the threshold, which means AUC does not change while the threshold changes. However, I'm getting different AUC values while changing the thresholds. I'm ...
- 399
0
votes
1
answer
55
views
Can I conclude that the classifier is always good when Precision-Recall Curve above the baseline?
I used logistic regression for highly imbalaned data (1=0.6% , 0=99.4%)
Since PR curves are sensitive to imbalance, so i used it, but I don't know how to interpret graph appropriately.
This is PR-...
0
votes
1
answer
108
views
how to caculate 95% CI for AUC? try 384 times or Hanley et al. (1982) method?
I am working on a prediction task to predict heart disease risk.
The data size is around 1500 and is splitted into train, validate and test datasets. I am use train dataset to train and use validate ...
0
votes
0
answers
41
views
How to calculate chance-level f1, ROC-AUC, PR-AUC for imbalanced dataset
I have an imbalance dataset (60% class 1, 40% class 0).
I trained a model and got accuracy, f1, ROC-AUC and PR-AUC.
I want to compare them to chance-level performance.
obviously chance-level of acc if ...
- 27
1
vote
2
answers
86
views
Given scores for a classifier and an expected positive rate, generate targets that achieve a given AUC score
Say I have output scores (maybe they're logits or poorly calibrated probabilities) from a trained classifier like y_hat = [1, 2, 3, 4], an expected positive rate of 25%, and I want to generate targets ...
- 1,172
0
votes
0
answers
191
views
1
vote
1
answer
273
views
Why would AUC on a validation set increase while loss increases?
I'm training a deep learning model in PyTorch. The first two images I posted here make perfect sense as they are the classical idea of overfitting. The training loss keeps decreasing while the ...
- 1,525
0
votes
0
answers
77
views
Correct SD for Area Under Curve (AUC)
I need to calculate the area under curve (AUC) with SD for several repeats of an experiment. I am using the trapeze method to estimate the area, where
$$
S = \sum(S_i)
$$
and
$$
S_i = (a+b)*d/2 = (f(...
- 23
2
votes
1
answer
219
views
Worse AUC but better metrics (Recall, Precision) on a classification problem - How can this happen?
I have two models on which I calculate train and test performances. They both are the same algorithm (lightgbm), same hyper-params, only the data differ (the second one has the data from the first one ...
