Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

Why PR score is down when balanced accuracy is good?

I just read this discussion here and here. I have a dataset of 977 records where class proportion is 77:23. My balanced accuracy is 75.5, ...
user avatar
  • 1,758
2 votes
2 answers
38 views

How is AUC helpful when we only need one threshold of a classifier

AUC is a summation of performance at different thresholds, but do we only care about a good performance at one threshold? Imagine a classifier with a low ROC but shots up at point of a low FP and high ...
user avatar
  • 21
0 votes
0 answers
15 views

Understand AUCs of different models

I'm testing two models against each other: provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %. provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
user avatar
  • 2,187
3 votes
1 answer
156 views

If the AUC score is 100 percent can F1 value be 99.94 percent?

If the AUC score is 100 percent can the F1 value be 99.94 percent? I would expect 100 percent, too.
user avatar
  • 189
0 votes
0 answers
9 views

Why R survAUC::AUC functions require both training and test data?

I am familiar with binary classification models, where the AUC is the metric from the area under the specificity-sensitivity curve, it indicates the performance on a dataset. Now, I'd like to assess ...
user avatar
  • 21
0 votes
0 answers
23 views

Make ROC graph with my problematic [duplicate]

For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic. I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
user avatar
0 votes
0 answers
13 views

Deriving Bayesian Credible Intervals for AUC using R brms

I am trying to estimate the posterior distribution for the AUC of a predictive biomarker using R brms. However, whenever I calculate the AUC using the posterior distribution of the model parameters, ...
user avatar
  • 43
0 votes
1 answer
23 views

How to compare multiple survival scores regarding its accuracy?

I have a clinical dataset which has already common prognosis scores, like characteristic tumor markers, tumor classification, etc. However I developed a new score which is able to predict patients' ...
user avatar
0 votes
1 answer
16 views

How to interpret this confusion matrix and roc curve?

I got these two images for evaluating a RF: I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %? By looking only at ...
user avatar
  • 2,187
0 votes
0 answers
14 views

AUC first drops before starting to grow

I have a very imbalanced classification problem, 99% vs 1%, and I am using logistic regression in pytorch. I often see the initial weights, randomly initialized, achieve AUC of 60-70% before any ...
user avatar
1 vote
0 answers
22 views

Can I use the AUR under the ROC on unbalanced test data?

I have split my data into training and test data, built several prediction models and now I want to evaluate the models using the test data set.The data is very unbalanced so I balanced the training ...
user avatar
  • 11
2 votes
1 answer
49 views

AUC - Logistic Regression versus LDA, and Naive Bayes

everyone! I am a newbie on machine learning, and I am now interested on classification modeling. I used logistic regression, linear discriminant analysis (LDA), and naive Bayes on my notebook DataCamp ...
user avatar
2 votes
1 answer
121 views

Appropriate way to get cross validated performance metrics

For cross-validation of a logistic regression classifier, it seems to me that there are a number of different approaches to calculating each performance metric: The performance metric is calculated ...
user avatar
3 votes
2 answers
59 views

Difference between ROC-AUC and Multiclass AUC (MAUC)

I am trying to understand the interpretation of these metrics in a multiclass scenario: ROC-AUC and MAUC. Scikit-learn provides ...
user avatar
  • 231
0 votes
0 answers
66 views

Comparing the difference in test AUC between different machine learning models using Delong's test

I'm in the process of developing four machine learning models (Neural network, logistic regression, XGBoost, SVM) on the same training data using the same exact features (10 features in my model) and ...
user avatar
1 vote
1 answer
52 views

AUC between estimated lines

Let's say you have a lmer model that test the drug effect of a set of rats with a set of rats (Control): lme1 <- lmer(lVolume ~ Days*Drug + (Days|Drug)) where ...
user avatar
0 votes
0 answers
23 views

Can anyone explain how to calculate AUC_DS, AUC_BW, and AUC_0 mentioned in this image?

I know AUC is for binary classification, but in a paper, the authors seemingly used AUC_DS, AUC_BW, and AUC_0 to compare vector values, like ground truth of one sample [3,2,1,0,1,2,3] prediction value ...
user avatar
  • 1
0 votes
0 answers
26 views

My data can be approximated with normal distribution (multimodal). How can I find the reasons and explain this behaviour?

I use DeLonge method to compare two ROC AUCS. The result of it is Z-score. Both ROC AUCs obtained from LDA (linear discriminant analysis) from sklearn package. The ...
user avatar
0 votes
0 answers
69 views

Precision-Recall curve and AUC in multi-class problems using R

I'm trying to evaluate a RandomForest model for multi-class classification using the Area Under Precision-Recall Curve. I need to plot the PR curve of each class and the micro and macro average and ...
user avatar
  • 113
0 votes
1 answer
66 views

How to report AUC from Cox Regression?

I need to calculate the AUC for a Cox regression model. SAS is giving me a time dependent AUC as below. How do I report this for the paper? Thanks!!
user avatar
  • 55
0 votes
0 answers
23 views

Feature Selection + Unpenalized Logistic Regression AUC

I am building a predictive model using a highly-dimensional dataset. To reduce dimensionality and reduce overfitting, I used LASSO regression to select features. ...
user avatar
0 votes
1 answer
20 views

Ridge regression coefficients show model importance but the model evaulation not

I have performed two ridge logistic regressions in R to check which of the two models perform better. From the first look of the coefficients, it looks like model1 ...
user avatar
  • 332
2 votes
1 answer
78 views

If going with the opposite prediction of a bad predictor gives good predictions, why not do that?

Let’s restrict our consideration to binary outcomes. I have a friend who is terrible at predicting the future, always predicting the opposite of what winds up happening. For instance, my friend ...
user avatar
  • 31k
0 votes
0 answers
31 views

How to compute ROC AUC for a method that uses two models?

To compare my method with others I'm trying to compute its AUC but I got a bit confused on how to do this for my case. My method uses a model that classifies an image as class A or B, after that if ...
user avatar
1 vote
2 answers
74 views

Proper way to finding area under ROC curve (AUC)

I have data of 200 persons with 2 variables: 1. C_level: Blood level of a chemical C (a numeric value) 2. Disease: Yes or No I want to know if C_level can be used ...
user avatar
  • 8,993
4 votes
4 answers
553 views

Validity of AUC for binary categorical variables

Scikit-learn function roc_auc_score can be used to get area under curve (AUC) of ROC curve. This score is generally used for numeric predictors' value in predicting ...
user avatar
  • 8,993
2 votes
1 answer
40 views

Is it valid to do ROC analysis without using test data?

I am trying to do an ROC analysis to check whether a particular biomarker has any diagnostic power for screening a condition. Since I have only small dataset of 100. I am not splitting the data into ...
user avatar
  • 811
0 votes
0 answers
17 views

Can you check the significance of your AUC values? [duplicate]

I have computed some AUC values from the ROC curve based on logistic regressions. Firstly, I have divided my two datasets (D1, D2...
user avatar
  • 332
0 votes
0 answers
30 views

ROC AUC for average and geometric mean of two models

Suppose I have two binary classification models built on the same dataset. One has ROC AUC of 0.86, another one has ROC AUC of 0.92. What will be the ROC AUC for the average of two models and their ...
user avatar
0 votes
1 answer
81 views

How to simulate a calibrated prediction model given prevalence and auc

I want to be able to simulate a prediction model given some prevalence of the event and the AUC of the model. I followed the method proposed here but, although this works for giving AUC and predicted ...
user avatar
0 votes
1 answer
17 views

AUC Instability when averaging over individual readers

I am trying to evaluated an algorithm that supports doctors when making a diagnosis. I have recruited 10 doctors. I have 50 training examples. Each doctor is randomly assigned 25 cases to review alone ...
user avatar
0 votes
0 answers
16 views

Does the imbalance in my data affect model quality in this case?

I have two datasets, used for classification task: training and validation one. Both of them are equally imbalanced, with about 24% of target value equal 1 and 75% of target value equal 0. I am using ...
user avatar
  • 113
0 votes
0 answers
41 views

AUC and misclassification rate [duplicate]

why do you think that is possible? I compare decision tree with neural network in SAS EM and it chooses DT as the best model, based on misclassification rate, I guess. But AUC is higher for neural ...
user avatar
  • 3
1 vote
0 answers
69 views

PRAUC curve Interpretation on imbalanced data

My training data is undersampled to a positive to negative ratio of 5%. I observe a PRAUC of .4 on my training data. When I test the model on real-world data where the positive to negative ratio is .5%...
user avatar
  • 21
2 votes
1 answer
20 views

Model selection in presence of overfitting - better test or closer train

Suppose I have a tree-based model (Random Forest for the sake of the example) and I play with a regularization parameter (tree depth) to fight overfitting. Eventually I can come up with two models - ...
user avatar
  • 318
1 vote
2 answers
143 views

Training AUC below 0.5

I've trained a logistic regression using a small number of predictors - pseudo R-squared is only 0.1 but I have significant terms and a nice low p value for the model. However, even on its own ...
user avatar
3 votes
1 answer
37 views

The different usage of ROC between diagnotic tests and machine learning

I am currently very confused about the ROC usage in diagnostic tests and machine learning. In the scenario of medical diagnostic studies, many tutorial does not mention the data split procedure as ...
user avatar
  • 43
3 votes
1 answer
41 views

Derivation of closed-form ROC expression under binormal assumptions

It's a known result that, under binormality assumptions, the area under the ROC curve (AUC) for a binary classifier has the following closed form. Formally, define the class conditional mean and ...
user avatar
  • 1,066
0 votes
0 answers
8 views

Is it possible to build an imputed 2-variable AUC model

I am building a 2-variable predictive model but there are some missing data for both of my predictors. Is it possible to combine MICE code with AUC code to get something called "imputed AUC model&...
user avatar
1 vote
0 answers
29 views

How to choose the balance between model fit vs AUC (diagnostic accuracy)?

I would like to know how we can choose between model fit (calibration) vs AUC when building the predictive model. For example, if I have one predictor which improves the model fit but results in a ...
user avatar
1 vote
1 answer
47 views

Relation between AUROC and threshold

As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that ...
user avatar
  • 481
0 votes
2 answers
197 views

AUC with different threshold

I know AUC is supposed to be independent on the threshold, which means AUC does not change while the threshold changes. However, I'm getting different AUC values while changing the thresholds. I'm ...
user avatar
0 votes
1 answer
55 views

Can I conclude that the classifier is always good when Precision-Recall Curve above the baseline?

I used logistic regression for highly imbalaned data (1=0.6% , 0=99.4%) Since PR curves are sensitive to imbalance, so i used it, but I don't know how to interpret graph appropriately. This is PR-...
user avatar
0 votes
1 answer
108 views

how to caculate 95% CI for AUC? try 384 times or Hanley et al. (1982) method?

I am working on a prediction task to predict heart disease risk. The data size is around 1500 and is splitted into train, validate and test datasets. I am use train dataset to train and use validate ...
user avatar
0 votes
0 answers
41 views

How to calculate chance-level f1, ROC-AUC, PR-AUC for imbalanced dataset

I have an imbalance dataset (60% class 1, 40% class 0). I trained a model and got accuracy, f1, ROC-AUC and PR-AUC. I want to compare them to chance-level performance. obviously chance-level of acc if ...
user avatar
  • 27
1 vote
2 answers
86 views

Given scores for a classifier and an expected positive rate, generate targets that achieve a given AUC score

Say I have output scores (maybe they're logits or poorly calibrated probabilities) from a trained classifier like y_hat = [1, 2, 3, 4], an expected positive rate of 25%, and I want to generate targets ...
user avatar
  • 1,172
0 votes
0 answers
191 views

Build ROC curve with only one class

I have three datasets structured as well: ...
user avatar
  • 115
1 vote
1 answer
273 views

Why would AUC on a validation set increase while loss increases?

I'm training a deep learning model in PyTorch. The first two images I posted here make perfect sense as they are the classical idea of overfitting. The training loss keeps decreasing while the ...
user avatar
  • 1,525
0 votes
0 answers
77 views

Correct SD for Area Under Curve (AUC)

I need to calculate the area under curve (AUC) with SD for several repeats of an experiment. I am using the trapeze method to estimate the area, where $$ S = \sum(S_i) $$ and $$ S_i = (a+b)*d/2 = (f(...
user avatar
2 votes
1 answer
219 views

Worse AUC but better metrics (Recall, Precision) on a classification problem - How can this happen?

I have two models on which I calculate train and test performances. They both are the same algorithm (lightgbm), same hyper-params, only the data differ (the second one has the data from the first one ...
user avatar

1
2 3 4 5
11