Questions tagged [roc]

Receiver Operating Characteristic, also known as the ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

What is the relationship between sensitivity/specificity and the ROC curve

I am testing a predictive program which return a range from 0 to 1. The higher the range the more likely the patient got the disease. I have a true or validated set of patients so I can test this ...
3 votes
1 answer
58 views

Why do we need to create a model when creating a ROC curve?

I do not have strong background in statistics but I believe I know the basics to understand what a ROC curve means. I have a table, first column with probabilities (from 0 to 1) from a predictive test ...
3 votes
1 answer
56 views

Is it normal for ROC curve thresholds to be Inf or -Inf?

I am running a binary classification with a random forest via the ranger package in R, and am using the ...
2 votes
1 answer
45 views

Is AUROC sample size-unbiased

Some metrics are sample size-biased, i.e., they will give biased estimates in small sample sizes. Is there a formal proof that AUROC is either sample size-biased or unbiased? I.e., will the ...
  • 1,751
0 votes
0 answers
27 views

Can I use a p-value as my decision threshold estimate when calculating AUC-ROC?

Sorry if the title is confusing. I essentially want to know if it's appropriate to use a p-value as the decision threshold (of the null outcome) when calculating AUC-ROC. I have a dataset of outcome ...
1 vote
0 answers
10 views

Best way to sum up and visualize the relation between results of 2 tests measuring the same outcome with 3 results

I have a dataset of different individuals who underwent 2 test that measure the same outcome with 3 possible results not reduced mildly reduced highly reduced I know that the distribution of the ...
  • 11
0 votes
0 answers
10 views

Reading and Interpreting two models performance from this ROC

This question is for educational goal. I trained a KNN with a public diabetes dataset If it was for developing a new diabetes test which model would you chose? Why? How should interpret the point ...
0 votes
0 answers
8 views

How to test multiple ROC-AUC of different models created by bootstrapping with each other for significant differences?

I have created two predictive models (binary classifiers) using caret in R. I used 500-fold bootstrapping as internal validation for this. This gives me 500 models per method. Similar to this: https://...
  • 51
0 votes
0 answers
11 views

Can someone explain Weighted ROC curves? and how do I chose the weights?

Thanks to this blog what is AUC, I got a clear understanding of AUC and how its calculated. However I fail to understand how are weighted AUC calculated? I have read somewhere that "weighted ROC ...
  • 101
0 votes
0 answers
32 views

Does the Precision-Recall AUC approach the ROC AUC as the data becomes balanced?

I am working on a Machine Learning classifier. It is a binary response and most predictor variables are categorical. I have several years of data and for some years, the response is imbalanced (more ...
  • 151
0 votes
1 answer
37 views

How to get an AUC plot for Cox Regression?

This is a figure for survival analysis on a dataset. The article is found at this link I am self teaching survival analysis. Up until recently, I believed that AUC and ROC were only for classification....
  • 391
1 vote
1 answer
129 views

Estimate optimal cutoff for time-dependent ROC

Recently I am working with survival data and trying to fit time-dependent ROC. And I have difficulty estimating the optimal cutoff for time-dependent ROC. Here is the process I do. ...
  • 35
1 vote
1 answer
49 views

How to get the threshold from PrecisionRecallDisplay?

My goal is to tune the Classifier with probability predict_proba() < threshold. Therefore, I need to get the threshold. The problem is ...
  • 203
1 vote
1 answer
66 views

How is it possible to get AUC = 0.72, but sensitivity = 0?

I've trained a Gradient Boost classifier and used it to predict a binary target variable. I plot the ROC, here it is, looks nice and good: However, using the same test data, this is the result of a ...
  • 423
1 vote
0 answers
30 views

Confusion regarding ROC

I am studying ROC for model evaluation for the first time and I have two questions that are confusing me : I totally understand how to construct ROC with a logistic regression model, in which for ...
  • 45
6 votes
1 answer
439 views

Is a 0.05 absolute difference in AUC values enough to declare statistically significant difference?

I have two binary classifiers working on the same dataset. If their ROC AUC values differ by at least 0.05, is it enough to declare that the performance difference between the two classifiers is ...
  • 187
0 votes
1 answer
31 views

How does one get from ROC curve to selecting the actual decision threshold of a classification model?

Edit to explain how this is different from the suggested duplicate: Reduce Classification Probability Threshold My question relates to the same topic, but is thoroughly different, so I'm surprised ...
  • 237
0 votes
0 answers
47 views

How to determine the ROC AUC score in sklearn for multi-class classification problems

In https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html, the multi-class ROC AUC score can be computed by multi_class='ovo' or ...
  • 989
0 votes
0 answers
40 views

Calculating the average ROC/AUC and variance

Background: I have calculated an ROC curve for 50 classifiers of the same type and want to combine all 50 curves into one plot (but do not wan to plot all 50). So I would like to find the average ROC ...
  • 1
0 votes
1 answer
49 views

Does it make sense to plot ROC curves on out-of-fold predictions in k-fold cross-validation?

To plot the ROC curve for the test set using k-fold cross-validation, the common approach is to plot k different ROC curves, one for each fold, and then obtain their average, as suggested in sklearn ...
  • 3,282
4 votes
1 answer
228 views

Smallest possible difference between AUC of two ranker

If there are 10 positive examples, and 90 negative examples in the test set, what is the smallest possible difference in AUC, between two rankers giving different AUC?
  • 305
4 votes
1 answer
75 views

Calculating Area Under Curve (AUC) using cumulative events and non-events rates after binning the data

I understand that the AUC is basically the area under the ROC curve, which is the plot of the proportion of true positives versus the proportion of false positives at different probability cutoffs. ...
  • 53
0 votes
0 answers
14 views

I optimized my K Value in a KNN model and it increased ROC however the model suffered in accuracy for the minority class

So I recently set out to build a KNN model and started with a KNN of 5. I received a ROC of .81 and accuracy for the majority and minority class of around .80. After I optimized K to 43 I assumed it ...
  • 1
1 vote
1 answer
33 views

What can be said about ROC of binary classifier trained for data with inverted class labels when we know ROC of classifier for original class labels?

In some domains we have negative/positive class well defined (assigned). In general binary classification problem we can assign class labels in two ways. When we have model trained for the original ...
1 vote
0 answers
88 views

What is the recommended goodness of fit test for binary logistic regression with large sample size?

I have a binary logistic regression with one unique independent variable X and several control variables. My data has over 240k observations. When I ran both HL and Pearson chi square goodness of fit ...
  • 143
0 votes
0 answers
51 views

Multiclass ROC curve about one vs one

I am learning ROC about multiclass problem. I read this article https://towardsdatascience.com/multiclass-classification-evaluation-with-roc-curves-and-roc-auc-294fd4617e3a . I am confused about ovo ...
2 votes
1 answer
192 views

Selecting best classification probability threshold with ROC/AUC doesn't necessarily improve F1 score

I read that probability based binary classifiers have 0.5 as default probability threshold for getting hard 0/1 labels (in scikit-learn for example) but this could be fine-tuned with methods like ...
  • 725
2 votes
3 answers
270 views

Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?

$AUC=0.5$ and an ROC curve of a 45-degree line often are considered the baseline performance of a model, one that gets absolutely nothing from the features. If we predict the same (prior) probability ...
  • 43.3k
0 votes
0 answers
9 views

how to get the diagnostic power of more than one variable combined?

I have three antibody measurements to predict a condition. I am doing ROC analysis for each of the antibodies to check what is the diagnostic power for each one. But I want to try what is the ...
  • 841
1 vote
0 answers
12 views

Does a model with 0.5 AUROC imply an average precision equal to the proportion of positive examples?

A random model has an area under the ROC curve equal to 0.5. We also know that a random model has an area under the Precision-Recall curve equal to the proportion (p) of positive examples. Then, here'...
  • 111
0 votes
0 answers
60 views

Calculating AUC from Match Score Matrix

I am reading "Introduction to Biometrics" (2011) by Jain, Ross, and Nandakumar. They propose the following scenario: Consider a scenario where the biometric data (e.g., right index ...
1 vote
1 answer
41 views

AUC-ROC interpretation

I performed AUC-ROC curve analysis to confirm the biomarker potential of genes that were significant in gene expression results. This is how the graphics came out in the genes I determined. How can I ...
  • 11
0 votes
1 answer
34 views

ROC curve illustrates a perfect classificator however other metrics show worse values

how is it to explain that ROC illustrates such a perfect classificator however other metrics represent something different? The evalaution was done on CV of 5. ...
  • 196
0 votes
1 answer
69 views

Is there something equivalent to a ROC curve for logistic GLMMs? [closed]

When I read previously about logistic regression, I recall some talk about ROC curves and how they can be good metrics for the sensitivity/specificity of a logistic regression as well as a goodness of ...
10 votes
2 answers
1k views

Is higher AUC always better?

Let's say we measure binary classifier performance by ROC graph, and we have two separate models with distinct AUC (The Area Under the Curve) values. Is the model with the higher AUC value always ...
  • 327
0 votes
0 answers
38 views

Is a Sensitivity-Specificity curve equal to a horizontally flipped ROC?

So I need to plot a Sensitivity-Specificity curve. Since ROCs represent TPR (sensitivity) against FPR (or 1 - Specificity), can I just plot TPR against 1 - FPR as in the code below to obtain a Sens-...
1 vote
0 answers
44 views

ROC curve - Question with exercise

I'm studying the ROC curve theory but I'm struggling with an apparently simple exercise. To recap what I know: "The ROC curve is a graphical plot that illustrates the diagnostic ability of a ...
1 vote
0 answers
38 views

Can't get AUC higher than 0.62 using XGBoost or Random forest [duplicate]

I have a dataframe with multiple numeric predictive features, and one binary target feature. 63% of the data belongs to class 0, and the rest to class ...
  • 423
0 votes
0 answers
23 views

does the ROC curve of a committee based predictor have any meaning?

would appreciate it if you'd take a moment to read the pipeline I've described below - it relates to how a learner that is based on a committee should be optimized w.r.t the threshold of the ROC curve....
  • 115
0 votes
0 answers
24 views

Evaluating country-level indicators through ROC analysis

Good afternoon. I have defined a binary classification problem. Classes are based on country-level data on child undernourishment whereby a positive class is assigned to a country whenever it ...
1 vote
0 answers
26 views

Interpreting the fitted model outputs from a covariate-adjusted ROC curve in the AROC package

Below I have fitted a covariate-adjusted ROC curve (AROC) using a semiparametric Bayesian normal linear regression model available in the R package “AROC”. However, I am not clear on how to interpret/...
1 vote
0 answers
55 views

Computing a Weighted AUC Metric

Consider the standard ROC-AUC metric, which has been shown to be equivalent to the normalized Wilcoxon-Mann-Whitney: $$ AUC = \frac{1}{N^{(+)}N^{(-)}} \sum_{i}\sum_{j}1[\hat{p}_i^{(+)} > \hat{p}_j^{...
  • 301
0 votes
2 answers
266 views

Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset

I am currently doing a logistic regression analysis and wanted to cross validate the results. Now that I have the probabillities calculated for each fold, I made a ROC Curve for each one as well. But ...
  • 43
1 vote
2 answers
84 views

Exchange the x-axis and y-axis in order to flip ROC curve

I have a continuous variable X and a binary response outcome D. The ROC curve I got for D with respect to X is below the diagonal line, as shown in the following picture. In this ROC curve, the true ...
  • 11
1 vote
0 answers
34 views

consistent gap between training and validation metrics

I am training a neural network (Deep and cross network) for a multi-label classification task (~700 labels). I am seeing a uniform gap between training and validation results on various metrics. E.g. ...
  • 311
2 votes
2 answers
38 views

Using a ROC Plot to interpret specific scores

I have a binary classifier which outputs a given score to differentiate normal (low score) from abnormal (high score) cases. The score itself however is non-interpretable to others. I know a ROC plot ...
1 vote
2 answers
27 views

Individual Measurement in Overlapping Distributions

I have a situation where I have measurements taken from samples of healthy and pathological populations. The pathological populations tend to have much higher measurements, but the distributions are ...
0 votes
0 answers
35 views

ROC curve representation in paper

Recently I crossed a paper, which authors presented their ROC curve results in the following way: 44,7,0,19,18,74,15,8,27,24,2,0,0,0,2,0,0,0,0,0,0,0, 0,0,0,183,42,36,46,59,25,6,2,2,15,330,81,59,86,1 ...
  • 407
4 votes
1 answer
324 views

how to compare the AUCs of ROC from nested models and non-nested models from published papers

I'm interested in comparing two AUCs from ROC in two nested models and two AUCs in non-nested models but using the same dataset in a paper. I know DeLong test can be used to compare them, but the ...
  • 143
0 votes
0 answers
22 views

Constant model AUC

In sklearn, if one uses a model that makes constant predictions (all predictions equal to a given value $x$), the ROC AUC value for that model is 0.5. However, I don't see that clearly from the area ...

1
2 3 4 5
18