Questions tagged [roc]
Receiver Operating Characteristic, also known as the ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system
860
questions
0
votes
0
answers
11
views
What is the relationship between sensitivity/specificity and the ROC curve
I am testing a predictive program which return a range from 0 to 1. The higher the range the more likely the patient got the disease.
I have a true or validated set of patients so I can test this ...
3
votes
1
answer
58
views
Why do we need to create a model when creating a ROC curve?
I do not have strong background in statistics but I believe I know the basics to understand what a ROC curve means.
I have a table, first column with probabilities (from 0 to 1) from a predictive test ...
3
votes
1
answer
56
views
Is it normal for ROC curve thresholds to be Inf or -Inf?
I am running a binary classification with a random forest via the ranger package in R, and am using the ...
2
votes
1
answer
45
views
Is AUROC sample size-unbiased
Some metrics are sample size-biased, i.e., they will give biased estimates in small sample sizes. Is there a formal proof that AUROC is either sample size-biased or unbiased? I.e., will the ...
- 1,751
0
votes
0
answers
27
views
Can I use a p-value as my decision threshold estimate when calculating AUC-ROC?
Sorry if the title is confusing. I essentially want to know if it's appropriate to use a p-value as the decision threshold (of the null outcome) when calculating AUC-ROC. I have a dataset of outcome ...
- 51
1
vote
0
answers
10
views
Best way to sum up and visualize the relation between results of 2 tests measuring the same outcome with 3 results
I have a dataset of different individuals who underwent 2 test that measure the same outcome with 3 possible results
not reduced
mildly reduced
highly reduced
I know that the distribution of the ...
- 11
0
votes
0
answers
10
views
Reading and Interpreting two models performance from this ROC
This question is for educational goal.
I trained a KNN with a public diabetes dataset
If it was for developing a new diabetes test which model would you chose? Why?
How should interpret the point ...
- 111
0
votes
0
answers
8
views
How to test multiple ROC-AUC of different models created by bootstrapping with each other for significant differences?
I have created two predictive models (binary classifiers) using caret in R. I used 500-fold bootstrapping as internal validation for this. This gives me 500 models per method. Similar to this:
https://...
- 51
0
votes
0
answers
11
views
Can someone explain Weighted ROC curves? and how do I chose the weights?
Thanks to this blog what is AUC, I got a clear understanding of AUC and how its calculated. However I fail to understand how are weighted AUC calculated?
I have read somewhere that "weighted ROC ...
- 101
0
votes
0
answers
32
views
Does the Precision-Recall AUC approach the ROC AUC as the data becomes balanced?
I am working on a Machine Learning classifier. It is a binary response and most predictor variables are categorical. I have several years of data and for some years, the response is imbalanced (more ...
- 151
0
votes
1
answer
37
views
How to get an AUC plot for Cox Regression?
This is a figure for survival analysis on a dataset. The article is found at this link
I am self teaching survival analysis. Up until recently, I believed that AUC and ROC were only for classification....
- 391
1
vote
1
answer
129
views
Estimate optimal cutoff for time-dependent ROC
Recently I am working with survival data and trying to fit time-dependent ROC.
And I have difficulty estimating the optimal cutoff for time-dependent ROC.
Here is the process I do.
...
- 35
1
vote
1
answer
49
views
How to get the threshold from PrecisionRecallDisplay?
My goal is to tune the Classifier with probability predict_proba() < threshold. Therefore, I need to get the threshold.
The problem is ...
- 203
1
vote
1
answer
66
views
How is it possible to get AUC = 0.72, but sensitivity = 0?
I've trained a Gradient Boost classifier and used it to predict a binary target variable. I plot the ROC, here it is, looks nice and good:
However, using the same test data, this is the result of a ...
- 423
1
vote
0
answers
30
views
Confusion regarding ROC
I am studying ROC for model evaluation for the first time and I have two questions that are confusing me :
I totally understand how to construct ROC with a logistic regression model, in which for ...
- 45
6
votes
1
answer
439
views
Is a 0.05 absolute difference in AUC values enough to declare statistically significant difference?
I have two binary classifiers working on the same dataset.
If their ROC AUC values differ by at least 0.05, is it enough to declare that the performance difference between the two classifiers is ...
- 187
0
votes
1
answer
31
views
How does one get from ROC curve to selecting the actual decision threshold of a classification model?
Edit to explain how this is different from the suggested duplicate: Reduce Classification Probability Threshold
My question relates to the same topic, but is thoroughly different, so I'm surprised ...
- 237
0
votes
0
answers
47
views
How to determine the ROC AUC score in sklearn for multi-class classification problems
In https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html, the multi-class ROC AUC score can be computed by multi_class='ovo' or ...
- 989
0
votes
0
answers
40
views
Calculating the average ROC/AUC and variance
Background: I have calculated an ROC curve for 50 classifiers of the same type and want to combine all 50 curves into one plot (but do not wan to plot all 50). So I would like to find the average ROC ...
- 1
0
votes
1
answer
49
views
Does it make sense to plot ROC curves on out-of-fold predictions in k-fold cross-validation?
To plot the ROC curve for the test set using k-fold cross-validation, the common approach is to plot k different ROC curves, one for each fold, and then obtain their average, as suggested in sklearn ...
- 3,282
4
votes
1
answer
228
views
Smallest possible difference between AUC of two ranker
If there are 10 positive examples, and 90 negative examples in the test set, what is the smallest possible difference in AUC, between two rankers giving different AUC?
- 305
4
votes
1
answer
75
views
Calculating Area Under Curve (AUC) using cumulative events and non-events rates after binning the data
I understand that the AUC is basically the area under the ROC curve, which is the plot of the proportion of true positives versus the proportion of false positives at different probability cutoffs. ...
- 53
0
votes
0
answers
14
views
I optimized my K Value in a KNN model and it increased ROC however the model suffered in accuracy for the minority class
So I recently set out to build a KNN model and started with a KNN of 5. I received a ROC of .81 and accuracy for the majority and minority class of around .80. After I optimized K to 43 I assumed it ...
- 1
1
vote
1
answer
33
views
What can be said about ROC of binary classifier trained for data with inverted class labels when we know ROC of classifier for original class labels?
In some domains we have negative/positive class well defined (assigned).
In general binary classification problem we can assign class labels in two ways.
When we have model trained for the original ...
- 47
1
vote
0
answers
88
views
What is the recommended goodness of fit test for binary logistic regression with large sample size?
I have a binary logistic regression with one unique independent variable X and several control variables. My data has over 240k observations. When I ran both HL and Pearson chi square goodness of fit ...
- 143
0
votes
0
answers
51
views
Multiclass ROC curve about one vs one
I am learning ROC about multiclass problem.
I read this article https://towardsdatascience.com/multiclass-classification-evaluation-with-roc-curves-and-roc-auc-294fd4617e3a .
I am confused about ovo ...
- 1
2
votes
1
answer
192
views
Selecting best classification probability threshold with ROC/AUC doesn't necessarily improve F1 score
I read that probability based binary classifiers have 0.5 as default probability threshold for getting hard 0/1 labels (in scikit-learn for example) but this could be fine-tuned with methods like ...
- 725
2
votes
3
answers
270
views
Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?
$AUC=0.5$ and an ROC curve of a 45-degree line often are considered the baseline performance of a model, one that gets absolutely nothing from the features.
If we predict the same (prior) probability ...
- 43.3k
0
votes
0
answers
9
views
how to get the diagnostic power of more than one variable combined?
I have three antibody measurements to predict a condition. I am doing ROC analysis for each of the antibodies to check what is the diagnostic power for each one. But I want to try what is the ...
- 841
1
vote
0
answers
12
views
Does a model with 0.5 AUROC imply an average precision equal to the proportion of positive examples?
A random model has an area under the ROC curve equal to 0.5.
We also know that a random model has an area under the Precision-Recall curve equal to the proportion (p) of positive examples.
Then, here'...
- 111
0
votes
0
answers
60
views
Calculating AUC from Match Score Matrix
I am reading "Introduction to Biometrics" (2011) by Jain, Ross, and Nandakumar. They propose the following scenario:
Consider a scenario where the biometric data (e.g., right index
...
- 111
1
vote
1
answer
41
views
AUC-ROC interpretation
I performed AUC-ROC curve analysis to confirm the biomarker potential of genes that were significant in gene expression results. This is how the graphics came out in the genes I determined. How can I ...
- 11
0
votes
1
answer
34
views
ROC curve illustrates a perfect classificator however other metrics show worse values
how is it to explain that ROC illustrates such a perfect classificator however other metrics represent something different?
The evalaution was done on CV of 5.
...
- 196
0
votes
1
answer
69
views
Is there something equivalent to a ROC curve for logistic GLMMs? [closed]
When I read previously about logistic regression, I recall some talk about ROC curves and how they can be good metrics for the sensitivity/specificity of a logistic regression as well as a goodness of ...
- 2,498
10
votes
2
answers
1k
views
Is higher AUC always better?
Let's say we measure binary classifier performance by ROC graph, and we have two separate models with distinct AUC (The Area Under the Curve) values. Is the model with the higher AUC value always ...
- 327
0
votes
0
answers
38
views
Is a Sensitivity-Specificity curve equal to a horizontally flipped ROC?
So I need to plot a Sensitivity-Specificity curve. Since ROCs represent TPR (sensitivity) against FPR (or 1 - Specificity), can I just plot TPR against 1 - FPR as in the code below to obtain a Sens-...
1
vote
0
answers
44
views
ROC curve - Question with exercise
I'm studying the ROC curve theory but I'm struggling with an apparently simple exercise.
To recap what I know:
"The ROC curve is a graphical plot that illustrates the diagnostic ability of a ...
- 11
1
vote
0
answers
38
views
Can't get AUC higher than 0.62 using XGBoost or Random forest [duplicate]
I have a dataframe with multiple numeric predictive features, and one binary target feature. 63% of the data belongs to class 0, and the rest to class ...
- 423
0
votes
0
answers
23
views
does the ROC curve of a committee based predictor have any meaning?
would appreciate it if you'd take a moment to read the pipeline I've described below - it relates to how a learner that is based on a committee should be optimized w.r.t the threshold of the ROC curve....
- 115
0
votes
0
answers
24
views
Evaluating country-level indicators through ROC analysis
Good afternoon.
I have defined a binary classification problem. Classes are based on country-level data on child undernourishment whereby a positive class is assigned to a country whenever it ...
- 1
1
vote
0
answers
26
views
Interpreting the fitted model outputs from a covariate-adjusted ROC curve in the AROC package
Below I have fitted a covariate-adjusted ROC curve (AROC) using a semiparametric Bayesian normal linear regression model available in the R package “AROC”. However, I am not clear on how to interpret/...
1
vote
0
answers
55
views
Computing a Weighted AUC Metric
Consider the standard ROC-AUC metric, which has been shown to be equivalent to the normalized Wilcoxon-Mann-Whitney:
$$
AUC = \frac{1}{N^{(+)}N^{(-)}} \sum_{i}\sum_{j}1[\hat{p}_i^{(+)} > \hat{p}_j^{...
- 301
0
votes
2
answers
266
views
Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset
I am currently doing a logistic regression analysis and wanted to cross validate the results. Now that I have the probabillities calculated for each fold, I made a ROC Curve for each one as well. But ...
- 43
1
vote
2
answers
84
views
Exchange the x-axis and y-axis in order to flip ROC curve
I have a continuous variable X and a binary response outcome D.
The ROC curve I got for D with respect to X is below the diagonal line, as shown in the following picture.
In this ROC curve, the true ...
- 11
1
vote
0
answers
34
views
consistent gap between training and validation metrics
I am training a neural network (Deep and cross network) for a multi-label classification task (~700 labels). I am seeing a uniform gap between training and validation results on various metrics. E.g. ...
- 311
2
votes
2
answers
38
views
Using a ROC Plot to interpret specific scores
I have a binary classifier which outputs a given score to differentiate normal (low score) from abnormal (high score) cases. The score itself however is non-interpretable to others.
I know a ROC plot ...
- 1,428
1
vote
2
answers
27
views
Individual Measurement in Overlapping Distributions
I have a situation where I have measurements taken from samples of healthy and pathological populations. The pathological populations tend to have much higher measurements, but the distributions are ...
- 1,428
0
votes
0
answers
35
views
ROC curve representation in paper
Recently I crossed a paper, which authors presented their ROC curve results in the following way:
44,7,0,19,18,74,15,8,27,24,2,0,0,0,2,0,0,0,0,0,0,0,
0,0,0,183,42,36,46,59,25,6,2,2,15,330,81,59,86,1
...
- 407
4
votes
1
answer
324
views
how to compare the AUCs of ROC from nested models and non-nested models from published papers
I'm interested in comparing two AUCs from ROC in two nested models and two AUCs in non-nested models but using the same dataset in a paper. I know DeLong test can be used to compare them, but the ...
- 143
0
votes
0
answers
22
views
Constant model AUC
In sklearn, if one uses a model that makes constant predictions (all predictions equal to a given value $x$), the ROC AUC value for that model is 0.5. However, I don't see that clearly from the area ...
- 209