Skip to main content

Questions tagged [roc]

Receiver Operating Characteristic, also known as the ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system at different thresholds

Filter by
Sorted by
Tagged with
3 votes
1 answer
111 views

Why would `pROC::roc` calculate $\max\{AUC, 1 - AUC\}$ by default?

There is some interesting behavior in the pROC::roc function in R. ...
Dave's user avatar
  • 65.1k
1 vote
0 answers
89 views

AUC > 0.5 under null model following feature selection

I've been going over the output of a Monte Carlo model that simulates disease risk as a function of genotype. Under a null model of no disease risk, we have 1000 case and 1000 control individuals. ...
Max's user avatar
  • 133
0 votes
0 answers
13 views

(How) can ROC/AUC analysis be adapted to problems with two (or more) thresholds?

(How) can ROC/AUC analysis be adapted to problems with two (or more) thresholds? Example 1: We measure variable $x$, assign label $1$, if it falls into interval $[a, b]$, and label $0$ otherwise. We ...
Roger V.'s user avatar
  • 4,439
0 votes
0 answers
14 views

Wanting to create a cut-off point for measurement (Length by width) at which an adverse outcome occurs

I am working on a project investigating dimensions of tears in shoulders and how it affects retear rate after an operation. From the literature, age, and tear size (Dimensions AP [width], ML [length], ...
Atham's user avatar
  • 1
0 votes
0 answers
7 views

How to draw a ROC curve given estimated probability that a unit is positive and actual observed class? [duplicate]

Assume that a classification model fitted to data available to you has provided for each statistical unit a probability $P(+|x)$ that the unit is positive. The following table shows all available ...
evans5's user avatar
  • 3
0 votes
0 answers
135 views

Using Predictive Value Confidence Intervals to "Predict" Outcomes

Here's the quick version: Say I have a confusion matrix with the following data based on a proficiency cut score on a pretest and outcomes on (passing/failing) a class. The cut score was determined ...
jacev123's user avatar
2 votes
1 answer
84 views

We have sensitivity-specificity space (ROC curves) and precision-recall space (PR curves, $F_1$ score). What work has been done with PPV-NPV space?

Receiver-operator characteristic (ROC) curves display the balance between sensitivity and specificity: how good you are at detecting category $1$ (sensitivity) while not falsely identifying category $...
Dave's user avatar
  • 65.1k
0 votes
0 answers
16 views

pROC plot shape with only one point

I've done this plot whit pROC ...
ArTu's user avatar
  • 213
0 votes
0 answers
11 views

"ROC AUC reflects the likelihood that a random positive instance will be located to the right of a random negative instance". How come? [duplicate]

According to this webpage, ROC AUC reflects the likelihood that a random positive (red) instance will be located to the right of a random negative (gray) instance. Would you please explain this ...
Evan Aad's user avatar
  • 1,443
0 votes
0 answers
20 views

Cut-off based on an ordinal variable in unbalanced panel data

I am currently looking for an appropriate statistical analysis for my research questions. I have a continuous variable (score) and an ordinal variable (test). Score is quadratically related to Test, i....
a.henrietty's user avatar
0 votes
0 answers
14 views

How is ROC AUC calculated for a Support Vector Machine?

My understanding is that a support vector machine (SVM) finds a hyperplane that separates two classes from each other. During training, there can be some amount of error allowed so that some classes ...
inquisitive_hamster's user avatar
0 votes
1 answer
19 views

AUC comparison with Delong when i.i.d is broken due to clustered data

I am asked to compare two AUC ROCs and output confidence interval for both. Delong method allows (and also Fast Delong) to have a stronger test than usual bootstrapping method. The data I am working ...
Tamsina Ludwig's user avatar
0 votes
0 answers
27 views

"Bi-bumped" ROC curve meaning?

I am exploring some models for the classification of proteins based on their similarity. Briefly, I am testing the classification with a binomial fit and then analyzing the corresponding ROC curve to ...
Natalia's user avatar
0 votes
0 answers
27 views

Can a ROC curve be a straight line even if the AUC is not 1.00? [duplicate]

I have a straight-line ROC curve even if the AUC is not 1.00, I do not understand if it is correct or not. Is it possible that the ROC curve is straight? Thanks in advance.
mirkojh's user avatar
0 votes
0 answers
36 views

How to Plot and Interpret the ROC Curve for Segmentation-based Object Detection Models?

I'm trying to plot the ROC Curve for a number of target/object detection models and compare their performance. The pre-trained models in question take an input image and they output a mask image where ...
Tungdil's user avatar
1 vote
0 answers
125 views

Using bootstrap to compare performance of two machine learning models

I have a binary classification problem, and two ML models $A$ and $B$. I evaluate performance of these models using the area under the ROC curve (AUROC). I want to assess whether model $A$ is ...
Jonathan's user avatar
  • 111
0 votes
0 answers
28 views

Compare average roc auc scores using T-Test

I am using xgbclassifier to do feature selection by removing variables based on feature importance. I checked the roc_auc_score through 10 cv, and got an average value of 0.785 when 5 features were ...
JAE's user avatar
  • 89
2 votes
1 answer
28 views

Relating population-level AUC (Somers's $D_{xy}$) to a mean shift

Say we have group $0$ distributed as $N(\mu, \sigma^2)$ and group $1$ distributed as $N(\mu+\delta, \sigma^2)$. We then use the Gaussian-distributed variables to predict group membership. It seems ...
Dave's user avatar
  • 65.1k
0 votes
0 answers
23 views

Using a Gain Curve to Evaluate Predictive Model and Determine Optimal Cutoff / Threshold / Decision Point for a Classification Problem

When building or evaluating a predictive model, we know that a ROC curve can be useful for identifying the optimal cutoff/threshold/decision point in a classification problem with a dichotomous ...
JamesFM's user avatar
1 vote
0 answers
16 views

When is PR curve more informative than ROC curve?

I am reading the paper A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks and the section 2 discusses the properties of AUROC vs AUPR. Some conclusions in the ...
longpollehn's user avatar
6 votes
1 answer
76 views

ROCAUC = average sensitivity across all thresholds according to IEEE TPAMI, yet my calculations show otherwise

Carrington et al (2023) make the claim that area under the receiver-operator characteristic curve is equal to the average sensitivity across all thresholds, and similarly for specificity (section 3), ...
Dave's user avatar
  • 65.1k
1 vote
1 answer
97 views

Integrated biomarkers for ROC

I'm am looking for ways to combine multiple quantitative values in order to build a ROC curve with specificity and sensitivity. This seems to be common in multiple biomarkers paper, but I can't find ...
Renaud's user avatar
  • 13
0 votes
0 answers
45 views

Classification Threshold varies wildly when using ROC curves for threshold moving

I'm trying to do threshold moving to get the appropriate threshold for an imbalanced dataset. I have a 1D timeseries that I am applying a binary transformer-based classifier on. I have: ...
Techie5879's user avatar
0 votes
0 answers
18 views

What are the main differences between a traditional receiver-operating curve and a Lorenz curve?

I am analyzing a dataset of cardiac echocardiographic exams, aiming to compare the diagnostic accuracy of a novel test in comparison to a standard one. I have recognized that using Stata I can ...
Giuseppe Biondi-Zoccai's user avatar
5 votes
1 answer
278 views

Is ROC curve unique?

ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter ...
Roger V.'s user avatar
  • 4,439
1 vote
1 answer
133 views

Why does my PR Curve look like this?

These are my recall and precision stats for the model I built. The Curve does not look good where recall is 0. Not sure why there are so many points there. Can anyone help and explain why the curve ...
ibarbo's user avatar
  • 65
1 vote
1 answer
141 views

Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?

It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
another_student's user avatar
3 votes
1 answer
142 views

What is the relationship between the Brier score "refinement" and the area under the ROC curve?

In the Wikipedia article on Brier score, there is a claim that the "refinement" in the two-component decomposition of Brier score is related to the area under the receiver-operator ...
Dave's user avatar
  • 65.1k
0 votes
0 answers
51 views

How does `sklearn.metrics.roc_curve` work without using model predictions? [duplicate]

I am trying to understand sklearn's function for computing the roc_curve. If I understand correctly, one needs the TPR and FPR to compute ROC. However, sklearn's function takes as input - ...
desert_ranger's user avatar
1 vote
0 answers
63 views

Am I able to compare pooled AUC values (from a meta-analysis) of two similar tests conducted in the same samples to establish which may be superior? [closed]

I'm new to all of this so I apologise in advance. I'm currently conducting a meta-analysis and have pooled AUCs that represent the accuracy of two very similar prognostic factors in predicting ...
Jim's user avatar
  • 11
4 votes
3 answers
985 views

Probability threshold in ROC curve analyses

When conducting a logistic regression analysis in SPSS, a default threshold of 0.5 is used for the classification table. Consequently, individuals with a predicted probability < 0.5 are assigned to ...
Manuel Leitner's user avatar
3 votes
2 answers
163 views

Calculate area under precision-recall curve from area under ROC curve and the prevalence

I am reading material that reports the area under a ROC curve. I am curious to know what the performance would be in precision-recall space. From the sensitivity and specificity values in the ROC ...
Dave's user avatar
  • 65.1k
1 vote
0 answers
14 views

Range of Subset of Population based on a Sample

Suppose I have a population with a known mean, range, and variance of a characteristic $v$. I am producing sets from this population that are of negligible size compared to the size of the population. ...
Mark McDermott's user avatar
2 votes
1 answer
353 views

Confidence Intervals of ROC Curve's AUCs overlap but delong test is significant?

I am using ci.auc in the pROC library to calculate AUC's confidence intervals and roc.test to calculate delong test. When I run the following: ...
Eddie's user avatar
  • 21
1 vote
0 answers
25 views

Are there any downsides to using AUROC curves in low event rate samples?

I was just asked to familiarize myself with some methods looking at comparing AUROC for a few predictive scores to predict outcomes. Issue is that I have a dataset of about 200 with <5% with the ...
Mike K's user avatar
  • 11
0 votes
0 answers
9 views

Different sensitivity and specificity for balanced populations

I am building a simple test to flag prehistorical hand prints as belonging men or women. Using a random variable built from the fingers lengths, and having measured a population, I built the ROC curve ...
Xavier Prudent's user avatar
0 votes
1 answer
396 views

Do ROC curves require probabilities?

In the binary case, the implementation of ROC curve in torchmetrics automatically applies a sigmoid when it detects logit inputs (i.e. when the values of scores are ...
Alex Bogatskiy's user avatar
0 votes
0 answers
37 views

ROC curve analysis for when having a training/validation/test split

I have a dataset I split in training/validation/testing data for a binary classification model. The data is used as following: Training data: for training the model (model weights, etc.) Validation ...
Alb's user avatar
  • 115
2 votes
1 answer
1k views

Delong's test for comparing the significance difference of two AUC

I have done two prediction model in R as an example as below: ...
user358238's user avatar
2 votes
3 answers
273 views

cutoff and auc and changing cutoff

can you tell me if this is ok? While the AUC (i.e. AUC of 0.6) we got is acceptable since it's bigger than 0.5, we may need to re-evaluate at our cutoff selections again. Because we can select cutoffs ...
Shawn Kim's user avatar
0 votes
0 answers
53 views

C-statistic (or AUC) for fractional logistic regression (i.e. continuous regression)

I have proportional data to which I have fit a logistic regression (i.e. fractional logistic regression). The statistician in our group wants me to provide a c-statistic for the regression. My ...
Brandon's user avatar
  • 356
3 votes
1 answer
193 views

Can a ROC curve be partly above and partly below the diagonal? [duplicate]

A ROC curve has particular meaning about how the sensitivity and specificity change as the classification threshold changes for two groups of data. The x- and y-axes both range from zero to one, and ...
Dave's user avatar
  • 65.1k
26 votes
6 answers
1k views

Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?

THE BOUNTY As promised, a bounty of $250$ points has been issued. A bounty-worthy answer should address the apparent controversy in the answers here that ROC curve interpretation does not depend on ...
Dave's user avatar
  • 65.1k
0 votes
0 answers
97 views

how to derive standard error for ROC and how to compare 2 ROC area under curve meta-analytic estimates in R using metamisc (Bayesian random effects)

I am trying to do a meta-analysis to pool many ROC curves together and get an estimate of cumulative AUC using 2 different ...
Mohamed Rahouma's user avatar
2 votes
0 answers
990 views

C-statistic vs AUC [closed]

I am analysing diagnostic accuracy. I have a dataset with a ground truth and 3 predictors. Ground truth = binary (0/1) Predictor 1-2 = binary (0/1) Predictor 3 = continuous (0-100) I have 50,000 ...
user12541161's user avatar
0 votes
0 answers
15 views

How does cumulative dynamic AUROC differ from traditional, binary-classification AUROC?

My goal is to predict components that are going to fail the soonest in order to replace them. To date, we've deployed a binary classification model, with the positive class representing a component's ...
boot-scootin's user avatar
2 votes
0 answers
281 views

Power/sample size estimation when comparing two AUCs (area under the curve)?

Are there any R functions (or other free software) for calculating power/sample size needed to compare 2 AUCs (area under the curve)? Specifically, suppose you want to fit 2 different models to a ...
arbetj's user avatar
  • 21
10 votes
3 answers
917 views

For a confusion matrix, is there a name for FP / (FP + FN)?

For a confusion matrix, there are a variety of useful rates, ratios and indices. But I cannot find the one I care about: FP / (FP + FN) Of course this measure is ...
David Bridgeland's user avatar
0 votes
1 answer
141 views

Model performance with multiply imputed data

I would like to know how to do calibration plot with Hosmer-lemeshow test and AUC for ROC curve after multiple imputation in R. I build one prediction model and tried to do model performance but ...
Haruka Hayashi's user avatar
2 votes
1 answer
100 views

Binary classification metrics - Combining sensitivity and specificity?

The harmonic mean between precision and recall (F1 score) is a common metric to evaluate binary classification. It is useful because it strikes a balance between precision (FP) and recall (FN). For ...
usual me's user avatar
  • 1,247

1
2 3 4 5
19