Questions tagged [roc]

Receiver Operating Characteristic, also known as the ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system

Filter by
Sorted by
Tagged with
1 vote
0 answers
56 views

Using bootstrap to compare performance of two machine learning models

I have a binary classification problem, and two ML models $A$ and $B$. I evaluate performance of these models using the area under the ROC curve (AUROC). I want to assess whether model $A$ is ...
Jonathan's user avatar
  • 111
0 votes
0 answers
15 views

Compare average roc auc scores using T-Test

I am using xgbclassifier to do feature selection by removing variables based on feature importance. I checked the roc_auc_score through 10 cv, and got an average value of 0.785 when 5 features were ...
JAE's user avatar
  • 9
2 votes
1 answer
23 views

Relating population-level AUC (Somers's $D_{xy}$) to a mean shift

Say we have group $0$ distributed as $N(\mu, \sigma^2)$ and group $1$ distributed as $N(\mu+\delta, \sigma^2)$. We then use the Gaussian-distributed variables to predict group membership. It seems ...
Dave's user avatar
  • 60.8k
0 votes
0 answers
12 views

Using a Gain Curve to Evaluate Predictive Model and Determine Optimal Cutoff / Threshold / Decision Point for a Classification Problem

When building or evaluating a predictive model, we know that a ROC curve can be useful for identifying the optimal cutoff/threshold/decision point in a classification problem with a dichotomous ...
JamesFM's user avatar
1 vote
0 answers
14 views

When is PR curve more informative than ROC curve?

I am reading the paper A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks and the section 2 discusses the properties of AUROC vs AUPR. Some conclusions in the ...
longpollehn's user avatar
6 votes
1 answer
65 views

ROCAUC = average sensitivity across all thresholds according to IEEE TPAMI, yet my calculations show otherwise

Carrington et al (2023) make the claim that area under the receiver-operator characteristic curve is equal to the average sensitivity across all thresholds, and similarly for specificity (section 3), ...
Dave's user avatar
  • 60.8k
1 vote
1 answer
35 views

How to determine "if the variables in the dataset distinguish between people with and without metabolic syndrome"? [duplicate]

I have a dataset of 18 variables (and 170 observations), contains 16 continuous variables. I have calculated the proportion of observations that have metabolic syndrome (19%). Now I have to determine &...
Nathan Vermaerke's user avatar
1 vote
1 answer
90 views

Integrated biomarkers for ROC

I'm am looking for ways to combine multiple quantitative values in order to build a ROC curve with specificity and sensitivity. This seems to be common in multiple biomarkers paper, but I can't find ...
Renaud's user avatar
  • 13
0 votes
0 answers
40 views

Classification Threshold varies wildly when using ROC curves for threshold moving

I'm trying to do threshold moving to get the appropriate threshold for an imbalanced dataset. I have a 1D timeseries that I am applying a binary transformer-based classifier on. I have: ...
Techie5879's user avatar
0 votes
0 answers
12 views

What are the main differences between a traditional receiver-operating curve and a Lorenz curve?

I am analyzing a dataset of cardiac echocardiographic exams, aiming to compare the diagnostic accuracy of a novel test in comparison to a standard one. I have recognized that using Stata I can ...
Giuseppe Biondi-Zoccai's user avatar
4 votes
1 answer
265 views

Is ROC curve unique?

ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter ...
Roger V.'s user avatar
  • 3,773
1 vote
1 answer
73 views

Why does my PR Curve look like this?

These are my recall and precision stats for the model I built. The Curve does not look good where recall is 0. Not sure why there are so many points there. Can anyone help and explain why the curve ...
ibarbo's user avatar
  • 35
1 vote
1 answer
50 views

Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?

It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
another_student's user avatar
0 votes
0 answers
40 views

What is the relationship between the Brier score "refinement" and the area under the ROC curve?

In the Wikipedia article on Brier score, there is a claim that the "refinement" in the two-component decomposition of Brier score is related to the area under the receiver-operator ...
Dave's user avatar
  • 60.8k
0 votes
0 answers
49 views

How does `sklearn.metrics.roc_curve` work without using model predictions? [duplicate]

I am trying to understand sklearn's function for computing the roc_curve. If I understand correctly, one needs the TPR and FPR to compute ROC. However, sklearn's function takes as input - ...
desert_ranger's user avatar
1 vote
0 answers
31 views

Am I able to compare pooled AUC values (from a meta-analysis) of two similar tests conducted in the same samples to establish which may be superior? [closed]

I'm new to all of this so I apologise in advance. I'm currently conducting a meta-analysis and have pooled AUCs that represent the accuracy of two very similar prognostic factors in predicting ...
Jim's user avatar
  • 11
4 votes
3 answers
465 views

Probability threshold in ROC curve analyses

When conducting a logistic regression analysis in SPSS, a default threshold of 0.5 is used for the classification table. Consequently, individuals with a predicted probability < 0.5 are assigned to ...
Manuel Leitner's user avatar
3 votes
2 answers
133 views

Calculate area under precision-recall curve from area under ROC curve and the prevalence

I am reading material that reports the area under a ROC curve. I am curious to know what the performance would be in precision-recall space. From the sensitivity and specificity values in the ROC ...
Dave's user avatar
  • 60.8k
1 vote
0 answers
11 views

Range of Subset of Population based on a Sample

Suppose I have a population with a known mean, range, and variance of a characteristic $v$. I am producing sets from this population that are of negligible size compared to the size of the population. ...
Mark McDermott's user avatar
2 votes
1 answer
123 views

Confidence Intervals of ROC Curve's AUCs overlap but delong test is significant?

I am using ci.auc in the pROC library to calculate AUC's confidence intervals and roc.test to calculate delong test. When I run the following: ...
Eddie's user avatar
  • 21
1 vote
0 answers
25 views

Are there any downsides to using AUROC curves in low event rate samples?

I was just asked to familiarize myself with some methods looking at comparing AUROC for a few predictive scores to predict outcomes. Issue is that I have a dataset of about 200 with <5% with the ...
Mike K's user avatar
  • 11
0 votes
0 answers
9 views

Different sensitivity and specificity for balanced populations

I am building a simple test to flag prehistorical hand prints as belonging men or women. Using a random variable built from the fingers lengths, and having measured a population, I built the ROC curve ...
Xavier Prudent's user avatar
0 votes
1 answer
181 views

Do ROC curves require probabilities?

In the binary case, the implementation of ROC curve in torchmetrics automatically applies a sigmoid when it detects logit inputs (i.e. when the values of scores are ...
level1807's user avatar
  • 101
0 votes
0 answers
30 views

ROC curve analysis for when having a training/validation/test split

I have a dataset I split in training/validation/testing data for a binary classification model. The data is used as following: Training data: for training the model (model weights, etc.) Validation ...
Alb's user avatar
  • 115
2 votes
1 answer
616 views

Delong's test for comparing the significance difference of two AUC

I have done two prediction model in R as an example as below: ...
user358238's user avatar
2 votes
3 answers
178 views

cutoff and auc and changing cutoff

can you tell me if this is ok? While the AUC (i.e. AUC of 0.6) we got is acceptable since it's bigger than 0.5, we may need to re-evaluate at our cutoff selections again. Because we can select cutoffs ...
Shawn Kim's user avatar
0 votes
0 answers
87 views

P-value for AUC of logistic regression model vs AUC=0.50, how?

In R studio, I am running a logistic regression model with a binary variable vs binary outcome. I get a AUC and a AUC confidence interval. Now my supervisor wants a p-value to show which auc are ...
DerekJones5's user avatar
0 votes
0 answers
32 views

C-statistic (or AUC) for fractional logistic regression (i.e. continuous regression)

I have proportional data to which I have fit a logistic regression (i.e. fractional logistic regression). The statistician in our group wants me to provide a c-statistic for the regression. My ...
Brandon's user avatar
  • 336
2 votes
1 answer
107 views

Can a ROC curve be partly above and partly below the diagonal? [duplicate]

A ROC curve has particular meaning about how the sensitivity and specificity change as the classification threshold changes for two groups of data. The x- and y-axes both range from zero to one, and ...
Dave's user avatar
  • 60.8k
25 votes
6 answers
744 views

Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?

THE BOUNTY As promised, a bounty of $250$ points has been issued. A bounty-worthy answer should address the apparent controversy in the answers here that ROC curve interpretation does not depend on ...
Dave's user avatar
  • 60.8k
0 votes
0 answers
59 views

how to derive standard error for ROC and how to compare 2 ROC area under curve meta-analytic estimates in R using metamisc (Bayesian random effects)

I am trying to do a meta-analysis to pool many ROC curves together and get an estimate of cumulative AUC using 2 different ...
Mohamed Rahouma's user avatar
1 vote
0 answers
390 views

C-statistic vs AUC [closed]

I am analysing diagnostic accuracy. I have a dataset with a ground truth and 3 predictors. Ground truth = binary (0/1) Predictor 1-2 = binary (0/1) Predictor 3 = continuous (0-100) I have 50,000 ...
user12541161's user avatar
0 votes
0 answers
14 views

How does cumulative dynamic AUROC differ from traditional, binary-classification AUROC?

My goal is to predict components that are going to fail the soonest in order to replace them. To date, we've deployed a binary classification model, with the positive class representing a component's ...
blacksite's user avatar
  • 646
2 votes
0 answers
198 views

Power/sample size estimation when comparing two AUCs (area under the curve)?

Are there any R functions (or other free software) for calculating power/sample size needed to compare 2 AUCs (area under the curve)? Specifically, suppose you want to fit 2 different models to a ...
arbetj's user avatar
  • 21
10 votes
3 answers
857 views

For a confusion matrix, is there a name for FP / (FP + FN)?

For a confusion matrix, there are a variety of useful rates, ratios and indices. But I cannot find the one I care about: FP / (FP + FN) Of course this measure is ...
David Bridgeland's user avatar
0 votes
1 answer
104 views

Model performance with multiply imputed data

I would like to know how to do calibration plot with Hosmer-lemeshow test and AUC for ROC curve after multiple imputation in R. I build one prediction model and tried to do model performance but ...
Haruka Hayashi's user avatar
2 votes
1 answer
77 views

Binary classification metrics - Combining sensitivity and specificity?

The harmonic mean between precision and recall (F1 score) is a common metric to evaluate binary classification. It is useful because it strikes a balance between precision (FP) and recall (FN). For ...
usual me's user avatar
  • 1,227
0 votes
0 answers
13 views

Is the StandardScaler important when using the logistic regression as a classifier? [duplicate]

I'm training my LR right now and using Brier Score and ROC AUC as my evaluation. In my x_train are binary variables, ordinal variables and numerical variables with a wide range (e.g. 20000 - 160000). ...
Luke's user avatar
  • 1
1 vote
6 answers
299 views

How to create optimal cut-off scores for a test placing students into different courses

Edit: Shared my solution as an answer here Our goal is to determine optimal cut-off test scores for course placement. The course placement has already been manually assigned to each test-taker. The ...
anneirb's user avatar
  • 71
0 votes
1 answer
28 views

Interpretation of ROC curve curving early

I am running a binary LASSO logistic regression using glmnet. The initial data I work with is raster spatial data. When I create an ROC (AUC ~ 0.72) curve based on the test data, the resulting curve ...
user avatar
2 votes
1 answer
181 views

ROC-AUC in GEE models?

I am wondering if it is possible to make ROC-AUC curves for GEE models? I found few papers who did that and it wasn't clear for me. I thought it was impossible given how they are marginal models. ...
Youknowme's user avatar
1 vote
0 answers
21 views

How to start GNN optimization to get higher precision?

I'm developing a GNN for missing links prediction following this blog post for PyG library. I'm using almost the same GNN with a different dataset. Altough my dataset is similar to the MovieLens ...
James's user avatar
  • 75
0 votes
1 answer
129 views

Are there any difference using scores or probabilities for roc_auc_score and precision_recall_curve functions?

I'm working with a GNN model for link prediction and using precision_recall_curve and roc_auc_score from the ...
James's user avatar
  • 75
0 votes
0 answers
34 views

AUC for Multi-Label Classification using SVM

I am tackling a multi-label classification problem and I want to choose a SVM model maximising the AUC. I am not sure if AUC can be used in this case and if yes it is sufficient just to change the ...
data_miner_frbg's user avatar
1 vote
0 answers
63 views

Combination of AUC for different subset of samples

Assuming I have ROC-AUC for multiple subsets of samples, is there a way to produce a "weighted" average of AUC or something that will represent/approximate the AUC for the complete ...
Gideon Kogan's user avatar
0 votes
0 answers
25 views

Accounting for spatial correlations in ROC analysis for image segmentation

Typically, we assume independent samples when performing ROC analysis. But for image data, e.g. in a segmentation problem, pixels in a neighborhood come from the same image and are spatially ...
Luis's user avatar
  • 1
0 votes
0 answers
21 views

How to measure uncertainty and account for spatial bias when conducting ROC analysis on image data?

I have a task where I need to perform ROC analysis and measure the AUC of the ROC curve, but the data is image data. I have pairs of images (which contain real-valued pixels) and masks (which contain ...
eigenlaplace's user avatar
1 vote
0 answers
213 views

Interpretation of area under the precision-recall curve

The area under the receiver-operator characteristic curve has a interpretation of how well the predictions of two categories are separated. This post gives the area under the precision-recall curve as ...
Dave's user avatar
  • 60.8k
8 votes
6 answers
2k views

Where in the ROC curve does it tell you what the threshold is?

In my understanding, the ROC curve plots the True positive rate and the False positive rate. However, I've also read in other places that the ROC curve helps determine where the threshold for ...
Katsu's user avatar
  • 861
2 votes
1 answer
243 views

ROC curve and thresholds: why does it never have the ideal point at the top left for observations close to certainty?

I am using ROC curves for multi-label classification. I have a classifier that produces a score for each label, say a Logistic Regression that produces a probability. I understand that an ROC curve is ...
miguelmorin's user avatar

1
2 3 4 5
19