Cross Validated
Sponsored by
Sponsored logo

Questions tagged [auc]

Ask Question

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

404 questions
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Feature Selection - Overfit?

I have a dataset of 100 patients and 1500 features. I split 80 train 20 test first and then use the train set to get the best hyperparameters / best feature subset doing the following: I randomly ...
1
vote
0answers
27 views

What is the appropriate use of the DeLong Test when comparing AUC of two different models tested on the same dataset?

In this scenario there are three datasets. Each dataset has 1000 images of an apple and 1000 images of a banana. Dataset A is used to train machine learning model A to classify whether the image is of ...
0
votes
1answer
27 views

Sampling distribution of AUC in multivariate normal distribution

I need to derive the sampling distribution of the above equation, which is sample mean of the AUC from what I see. The first picture describes the distribution of the data, which I understand is ...
0
votes
0answers
9 views

Precision and recall for SVM from Confusion matrix is different from Precision-Recall graph

Coming from Stackoverflow- So, I am creating a SVM model for a highly imbalanced data set and trying to create to calculate F, Pression and recall from the confusion matrix of the model. Confusion ...
2
votes
0answers
24 views

Odd results of AUC and McFadden R2 on insure tech article

I was reading this article entitled: Usage-Based Vehicle Insurance: Driving Style Factors of Accident Probability and Severity (Korishchenko et all., 2019) [1], and watching the results section, I saw ...
0
votes
0answers
15 views

Is it normal to have a high AUC in train test

I have Random Forest classification model which is already tunned using k-folds cross validation, when I train the model in the train set, the auc gives me 0.97, in test set is 0.75. Is it normal to ...
0
votes
0answers
30 views

Why 1 out of 5 AUC cross validation score is very low?

I am using xgboost model for binary classification problem. I am using 5 fold cross-validation (stratified as class imbalance) which results into the following. ...
0
votes
0answers
18 views

Connection between prob output LogisticReg/SVM and ROC

I have the following ROC generated using LPOCV and Logistic regression or SVM (l2 norm). Now, let's say I have a test set containing 10 patients and I get that the probabilities of those patients to ...
0
votes
0answers
26 views

AUC's standard deviation for a 10 x 10-cross validation test

In the following thesis http://arno.uvt.nl/show.cgi?fid=147278 the user compute the AUC standard deviation as measure of robustness. Let's say I have run a repeated (10) 10-cross validation ...
0
votes
0answers
9 views

How to calculate the standard error of the area under the curve (AUC) of prognostic studies with 95% C.I

I have to do a meta-analysis of the area under the curve (AUC) of prognostic models. The AUC and 95% confidence intervals have been provided. I need the standard error of the AUC to proceed. How do I ...
1
vote
1answer
27 views

Imbalanced dataset - Majority positive class

My dataset consists of 150 patients where 50 are controls/healthy (negative) and 100 are sick (positive). If I want my model to have high sensitivity at high specificity (left side of the ROC), in ...
1
vote
1answer
38 views

AUC ROC and probabilistic interpretation

I can't solve the problem about the AUC ROC metric. Problem condition: on the answers (estimates) of the algorithm, objects of class 0 are distributed uniformly on the segment [0, 2/3], and answers of ...
1
vote
0answers
73 views

Estimating how easy it is to classify each individual observation

Is there a way of quantifying something like an AUC per observation. What I'm trying to estimate is how easy each observation is to classify. So for example, given a test set $X$ of size $m$ ...
0
votes
0answers
15 views

AUC for Repeted 10-fold cross validations

I'm performing repeated 10-fold cross-validation in order to validate the predictive accuracy of an ML model. the following paper https://www.hpl.hp.com/techreports/2009/HPL-2009-359.pdf is ...
1
vote
1answer
33 views

Comparing multiclass AUC between two classifiers trained on the same data

I have fit two different multiclass classifiers on the same data, one is a multinomial logistic regression the other one a KNN Classifier. Now I want to know which one has a better fit in general. I ...
1
vote
0answers
18 views

Relationship between standard error and confidence interval of AUC

I apologize if this is a very basic question. When I see a term like AUC 0.72 ± 0.02, what does the number after ± refer to? Is this the standard error? Secondly, how can I find the 95% confidence ...
2
votes
1answer
34 views

AUROC: Regression vs Classification results

I'm trying to evaluate the probability of a rare occurrence. My training data is a binary 1/0 for output and a TFIDF Vector of words for input. Which seems to lend itself to a regression, I've been ...
0
votes
1answer
27 views

Stable model or overfitting?

I have a dataset of 150 patients (2:1 ratio of classes) and 78 features. I performed backwards elimination using logistic regression feature importance to end up with 13 features (SVC classifier). I ...
1
vote
0answers
26 views

How many datapoints needed to reduce AUC confidence intervals?

I am using LOOCV to calculate AUC and using the bootstrap BCa approach to calculate the confidence intervals. Now, I've read that the confidence intervals drop by root(N), is N the number of ...
0
votes
0answers
20 views

Feature selection to calculate AUC using the LOOCV approach

I came across this paper: https://www.ncbi.nlm.nih.gov/pubmed/29355115 where the authors use random forests for feature selection in the following way: "..we performed the RFE procedure 100 times ...
1
vote
0answers
40 views

Will oversampling help with generalization (small imbalanced dataset)?

I have an imbalanced dataset (2:1 ratio) with about 60 patients and 80 features. I performed RFE + stratified cross validation to reduce the features to 15 and I get an AUC of 0.9 with Logistic ...
1
vote
1answer
28 views

ROC curves: can a cut-point (cut-off) be “useful”, or is it a term reserved to parameters only?

This is a semantic question: When analyzing Receiver Operating Characteric (ROC) curves, we talk about the "usefulness" of a parameter based on its Area Under the Curve (AUC). "Useful" here refers to ...
0
votes
0answers
9 views

sample size cohort prospective

I doing my final research with main topic to compare AUC (area under the curve) between modified GRACE SCORE with the original GRACE Score. which equation should I used to calculate sample size for ...
0
votes
1answer
23 views

Is AUC for binary predictor equal to balanced accuracy?

... because they come up the same when I compute them for binary predictions. So either they are the same, or I have something wrongly implemented.
0
votes
1answer
29 views

The precision recall AUCs calculated by two different packages are different?

I used the dataset cars as an example ...
1
vote
2answers
58 views

Why is the mean of sensitivity and specificity equal to the AUC?

For a given cut-point in a prediction model or score, the mean of sensitivity and specificity equals the AUC. I've read that and I have observed this empirically. How can I prove this?
0
votes
2answers
81 views

Predictive performance of joint models versus standard survival models

I am trying to show that predictions based on repeated measures of markers (using joint modelling of repeated markers and time to event models: JMbayes package) are better than those based on only one ...
5
votes
1answer
96 views

What is the relationship between the Harrel's C and the AUC?

I model survival outcomes using a Cox proportional hazards model and want to evaluate model fit. Harrel's concordance index C is defined as the proportion of observations that the model can order ...
0
votes
0answers
9 views

Left-Truncated Right-Censored Cox Proportional Hazard Regression

If I have a left-truncated (remove everyone who has disease at baseline) and right-censored (1 if the person develops disease, 0 else). Time to event = min(age censored, age at disease diagnosis) ...
1
vote
0answers
17 views

Unrealistically high AUC-ROC score comparing to control feature and other performance measures

I am making a binary classification using regularized logistic regression, with extreme unbalanced data. The target label is Tar and non-target label is ...
0
votes
2answers
97 views

AUC or $R^2$/RMSE for binary classification

I am using doing a binary classification to classify things 0 or 1 using a set of features with LightGBM and XGBoost. Both models give AUC scores roughly in the <...
0
votes
0answers
49 views

Comparing AUC, logloss and accuracy scores between models

I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem: ...
1
vote
1answer
52 views

How to calculate and plot cross-validated ROC?

I am doing K-fold cross validation and I want to plot an averaged ROC curve in MATLAB. However currently I can only plot K ROC curves in one plot but without knowing the algorithm of averaged ROC ...
1
vote
1answer
33 views

Area under the curve in risk prediction model

The area under the curve for my data set is $0.63$. However, when I divided my data randomly into two parts, development (67%) and validation (33%), the value of the area under the curve became $0.58$...
0
votes
0answers
61 views

Why Log loss, AUC and precision & recall change differently when class imbalance problem is tackled?

I have a dataset and I'm working on a binary classification task with it. It has a class imbalance problem where False class versus ...
3
votes
0answers
48 views

Power and sample size calculations for unpaired ROC curves

I would like to perform power and sample size calculations for comparison of unpaired receiver-operating characteristic (ROC) curves. I have tried using the power.roc.test function from pROC package ...
-1
votes
1answer
31 views

R-Caret : Not-meaningful class probabilities and AUC value

I am very new to ML therefore my question might be primitive. I am working on a binary-class problem. The response (target) variable is occurrence : a factor ...
1
vote
1answer
79 views

performance measure suited for imbalanced classes and robust towards changing class ratios

I am looking for the best performance measure. My use case: I want to find out which dataset can be modelled best with binary classification. The datasets have an active minority class I am ...
0
votes
0answers
16 views

Is it necessary to report the training AUC?

When developing a classification model, is it mandatory to report the AUC for the training data (i.e. report the AUC for each fold of k-fold cross-validation)? I am studying how changing the number of ...
0
votes
1answer
41 views

Calculate AUC using predicted values and labels from a 5 fold classification?

I have a classifier for a binary problem. That has outputs between 0 and 1 for predictions for the two class A or B (for example sunny, not sunny). The classifier has ran on 5 unique folds of the data ...
0
votes
1answer
59 views

Confidence Interval of AUC ROC more than 1?

I'm analyzing a biomarker, and the Sens and Spec look promising (>90). however, the AUROC confidence interval is (0.929 to 1.016). I have a small sample size, however I would like to know how to ...
1
vote
0answers
32 views

When do ROC-Curves intersect?

In books I see ROC curves like this: So I always think the model with the highest AUC is better at any threshold. Is this true in general? When applying usual classification models, can there be ...
0
votes
2answers
464 views

Calculate AUC using sensitivity and specificity values only

How to calculate AUC, if I have values of sensitivity and specificity for various threshold cutoffs? I have sensitivity and specificity values for 100 thresholds. ...
3
votes
2answers
233 views

PR AUC < 50% with ROC AUC > 90% - model good or bad?

I understand for highly imbalanced dataset - we need to look for precision-recall vs ROC AUC to better judge the model. My question is what is the range for PR AUC below which the model is bad? My ...
1
vote
0answers
18 views

AUC-like measure for multiple simultaneous classification tasks?

I know that given an ordered set of binary labels, and equally-sized ordered set of scalar predictions, we can quantify how cleanly the predictions separate the labels into clean buckets of 0's and 1'...
2
votes
1answer
34 views

High variation of AUC score when fitting a logistic regression model

I'm using sklearn LogisticRegression with a training data set of 279 inputs. Each input point belongs to $[0,1]^2$ and to a class. There are two classes: $\{0, 1\}$...
2
votes
1answer
130 views

Generate synthetic data given AUC

I'm experimenting with ROC-AUC for binary classification problems. I want to generate synthetic data for a given AUC score. The ...
13
votes
1answer
278 views

Does a logistic regression maximizing likelihood necessarily also maximize AUC over linear models?

Given a data set with binary outcomes $y\in\{0,1\}^n$ and some predictor matrix $X\in\mathbb{R}^{n\times p}$, the standard logistic regression model estimates coefficients $\beta_{MLE}$ which maximize ...
0
votes
1answer
65 views

Can a relationship between x and y be modeled, if all the data points fill the area under a curve?

I'd like to derive an equation that enables me to calculate y based on x. I'm having troubles figuring out how to do this as my data doesn't form a line/curve, but rather creates an edge (image below)...
0
votes
2answers
93 views

Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?

My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking: Please provide ...

1 2 3 4 5 … 9