Questions tagged [roc]
Receiver Operating Characteristic, also known as the ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system
849
questions
1
vote
1
answer
14
views
Selecting best classification probability threshold with ROC/AUC doesn't necessarily improve F1 score
I read that probability based binary classifiers have 0.5 as default probability threshold for getting hard 0/1 labels (in scikit-learn for example) but this could be fine-tuned with methods like ...
- 715
2
votes
2
answers
82
views
Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?
$AUC=0.5$ and an ROC curve of a 45-degree line often are considered the baseline performance of a model, one that gets absolutely nothing from the features.
If we predict the same (prior) probability ...
- 35.8k
0
votes
0
answers
9
views
how to get the diagnostic power of more than one variable combined?
I have three antibody measurements to predict a condition. I am doing ROC analysis for each of the antibodies to check what is the diagnostic power for each one. But I want to try what is the ...
- 841
1
vote
0
answers
11
views
Does a model with 0.5 AUROC imply an average precision equal to the proportion of positive examples?
A random model has an area under the ROC curve equal to 0.5.
We also know that a random model has an area under the Precision-Recall curve equal to the proportion (p) of positive examples.
Then, here'...
- 111
0
votes
0
answers
11
views
Calculating AUC from Match Score Matrix
I am reading "Introduction to Biometrics" (2011) by Jain, Ross, and Nandakumar. They propose the following scenario:
Consider a scenario where the biometric data (e.g., right index
...
- 111
1
vote
1
answer
30
views
AUC-ROC interpretation
I performed AUC-ROC curve analysis to confirm the biomarker potential of genes that were significant in gene expression results. This is how the graphics came out in the genes I determined. How can I ...
- 11
0
votes
1
answer
23
views
ROC curve illustrates a perfect classificator however other metrics show worse values
how is it to explain that ROC illustrates such a perfect classificator however other metrics represent something different?
The evalaution was done on CV of 5.
...
- 196
0
votes
1
answer
29
views
Is there something equivalent to a ROC curve for logistic GLMMs? [closed]
When I read previously about logistic regression, I recall some talk about ROC curves and how they can be good metrics for the sensitivity/specificity of a logistic regression as well as a goodness of ...
10
votes
2
answers
780
views
Is higher AUC always better?
Let's say we measure binary classifier performance by ROC graph, and we have two separate models with distinct AUC (The Area Under the Curve) values. Is the model with the higher AUC value always ...
- 317
0
votes
0
answers
23
views
Is a Sensitivity-Specificity curve equal to a horizontally flipped ROC?
So I need to plot a Sensitivity-Specificity curve. Since ROCs represent TPR (sensitivity) against FPR (or 1 - Specificity), can I just plot TPR against 1 - FPR as in the code below to obtain a Sens-...
1
vote
0
answers
34
views
ROC curve - Question with exercise
I'm studying the ROC curve theory but I'm struggling with an apparently simple exercise.
To recap what I know:
"The ROC curve is a graphical plot that illustrates the diagnostic ability of a ...
- 11
1
vote
0
answers
37
views
Can't get AUC higher than 0.62 using XGBoost or Random forest [duplicate]
I have a dataframe with multiple numeric predictive features, and one binary target feature. 63% of the data belongs to class 0, and the rest to class ...
- 153
0
votes
0
answers
17
views
does the ROC curve of a committee based predictor have any meaning?
would appreciate it if you'd take a moment to read the pipeline I've described below - it relates to how a learner that is based on a committee should be optimized w.r.t the threshold of the ROC curve....
- 115
0
votes
0
answers
23
views
Evaluating country-level indicators through ROC analysis
Good afternoon.
I have defined a binary classification problem. Classes are based on country-level data on child undernourishment whereby a positive class is assigned to a country whenever it ...
- 1
1
vote
0
answers
14
views
Interpreting the fitted model outputs from a covariate-adjusted ROC curve in the AROC package
Below I have fitted a covariate-adjusted ROC curve (AROC) using a semiparametric Bayesian normal linear regression model available in the R package “AROC”. However, I am not clear on how to interpret/...
1
vote
0
answers
23
views
Computing a Weighted AUC Metric
Consider the standard ROC-AUC metric, which has been shown to be equivalent to the normalized Wilcoxon-Mann-Whitney:
$$
AUC = \frac{1}{N^{(+)}N^{(-)}} \sum_{i}\sum_{j}1[\hat{p}_i^{(+)} > \hat{p}_j^{...
- 281
0
votes
2
answers
53
views
Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset
I am currently doing a logistic regression analysis and wanted to cross validate the results. Now that I have the probabillities calculated for each fold, I made a ROC Curve for each one as well. But ...
- 43
1
vote
2
answers
45
views
Exchange the x-axis and y-axis in order to flip ROC curve
I have a continuous variable X and a binary response outcome D.
The ROC curve I got for D with respect to X is below the diagonal line, as shown in the following picture.
In this ROC curve, the true ...
- 11
1
vote
0
answers
17
views
consistent gap between training and validation metrics
I am training a neural network (Deep and cross network) for a multi-label classification task (~700 labels). I am seeing a uniform gap between training and validation results on various metrics. E.g. ...
- 311
1
vote
2
answers
30
views
Using a ROC Plot to interpret specific scores
I have a binary classifier which outputs a given score to differentiate normal (low score) from abnormal (high score) cases. The score itself however is non-interpretable to others.
I know a ROC plot ...
- 1,248
0
votes
2
answers
22
views
Individual Measurement in Overlapping Distributions
I have a situation where I have measurements taken from samples of healthy and pathological populations. The pathological populations tend to have much higher measurements, but the distributions are ...
- 1,248
0
votes
0
answers
32
views
ROC curve representation in paper
Recently I crossed a paper, which authors presented their ROC curve results in the following way:
44,7,0,19,18,74,15,8,27,24,2,0,0,0,2,0,0,0,0,0,0,0,
0,0,0,183,42,36,46,59,25,6,2,2,15,330,81,59,86,1
...
- 405
4
votes
1
answer
254
views
how to compare the AUCs of ROC from nested models and non-nested models from published papers
I'm interested in comparing two AUCs from ROC in two nested models and two AUCs in non-nested models but using the same dataset in a paper. I know DeLong test can be used to compare them, but the ...
- 143
0
votes
0
answers
18
views
Constant model AUC
In sklearn, if one uses a model that makes constant predictions (all predictions equal to a given value $x$), the ROC AUC value for that model is 0.5. However, I don't see that clearly from the area ...
- 157
1
vote
0
answers
47
views
Ties in AUC probabilistic definition
According to the ROC AUC definition in here:
The correct statement is that ROC AUC is the probability a randomly-chosen positive example is ranked more highly than a randomly-chosen negative example.
...
- 157
1
vote
0
answers
36
views
How to interpret ROC curve if my FPR (X-axis) values ranges from 0 to 0.02?
I am plotting the ROC curve for my classification problem. The results I am getting for the problem are TPR ranges from 0 to 1, but the FPR ranges from 0 to 0.02. I have plotted the ROC curve by ...
- 33
2
votes
1
answer
31
views
Examples of problems where ROC curves are more useful than PR curves
I understand the formal differences between a ROC curve and a PR curve, but I am trying to build more intuition. For common applications such as Search and disease prediction, PR curves are more ...
- 995
1
vote
1
answer
25
views
Can the AUC of the ROC be interpreted as the average recall?
The PR curve plots the precision as a function of the recall, and the AUC can be interpreted as the average precision.
The ROC plots the recall as a function of the specificity. Can we interpret the ...
- 995
0
votes
1
answer
25
views
When should I balance my data using AUROC and AUPRC?
I want to report the AUROC and the AUPRC of a prediction model using an unbalanced dataset. Is it correct that I have to balance my data to calculate the AUROC but leave the data unbalanced to ...
- 21
1
vote
0
answers
14
views
AUROC or AUPRC? [duplicate]
I have a highly imbalanced dataset and want to evaluate the performance of a score. Is it better to balance the data and calculate the area under the receiver operating characteristic or to use the ...
- 21
0
votes
0
answers
77
views
Confusion matrix vs ROC-AUC curve
I'm using the lares packages in combination with H2O and with
...
- 2,441
2
votes
2
answers
62
views
How is AUC helpful when we only need one threshold of a classifier
AUC is a summation of performance at different thresholds, but do we only care about a good performance at one threshold?
Imagine a classifier with a low ROC but shots up at point of a low FP and high ...
- 21
0
votes
0
answers
13
views
Area Under Precision-Recall and Area Under ROC curve for different amount of observations
I am doing a research and thus comparing some algorithms for binary classification. Worth to mention that, the data set is highly imbalanced i.e., the minority class is only 0.2%.
Notation:
Area Under ...
- 1
0
votes
0
answers
17
views
Understand AUCs of different models
I'm testing two models against each other:
provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %.
provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
- 2,441
0
votes
0
answers
22
views
what is the acceptable value for acceptable cut point while minimizes distance between ROC plot and point (0,1)
If we would like to find the optimal cut point using minimizes distance between ROC plot and point (0,1), is it a acceptable value in the field for the roc01 value? I have searched for the reference ...
- 1
0
votes
0
answers
32
views
What is my best option to evaluate a predictive (or proxy) variable?
I have a list of thunder flashes, and I'm trying to find if a meteorological variable (CAPE) is a good predictable variable (or proxy) for theses flashes. In my thoughts, I want to evaluate this proxy ...
- 1
0
votes
0
answers
23
views
Make ROC graph with my problematic [duplicate]
For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic.
I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
- 1
0
votes
0
answers
45
views
Determining cut-off (Youden) without causing data leakage
I´m new to coding and machine learning so this might sound like a stupid question.
I´m using Logistic regression, RF and SVM to model corporate defaults in R. However i´m a bit concerned of what could ...
- 1
0
votes
1
answer
31
views
How to interpret this confusion matrix and roc curve?
I got these two images for evaluating a RF:
I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %?
By looking only at ...
- 2,441
2
votes
1
answer
78
views
AUC - Logistic Regression versus LDA, and Naive Bayes
everyone!
I am a newbie on machine learning, and I am now interested on classification modeling.
I used logistic regression, linear discriminant analysis (LDA), and naive Bayes on my notebook DataCamp ...
- 33
0
votes
0
answers
9
views
Is there any Statistic method which could reflect the diagnose value when the prevalence of special type of characteristic is lower
Our present study is conducting a novel diagnosis test for predicting a disease by the X-ray. The gold-standard is a minimally invasive procedure can test a tissue sample for the disease.
In our ...
1
vote
1
answer
228
views
Logistic Regression: What is the value for precision when recall (true positive rate) is 0?
A quick overview of definitions before I get into the question:
True Positive (TP): An actual positive that the model classified as positive
False Positive (FP): An actual negative that the model ...
- 211
1
vote
1
answer
54
views
Is there a good reason for using AUROC on imbalanced dataset?
So I just learned about AUROC.
When I read this thread, it seems like AUROC is not a great metric for imbalanced dataset. One answer even says it shouldn't be used to compare models.
However, I am ...
- 135
3
votes
2
answers
351
views
Difference between ROC-AUC and Multiclass AUC (MAUC)
I am trying to understand the interpretation of these metrics in a multiclass scenario: ROC-AUC and MAUC. Scikit-learn provides ...
- 231
0
votes
0
answers
79
views
My data can be approximated with normal distribution (multimodal). How can I find the reasons and explain this behaviour?
I use DeLonge method to compare two ROC AUCS. The result of it is Z-score.
Both ROC AUCs obtained from LDA (linear discriminant analysis) from sklearn package. The ...
- 1
0
votes
0
answers
255
views
Precision-Recall curve and AUC in multi-class problems using R
I'm trying to evaluate a RandomForest model for multi-class classification using the Area Under Precision-Recall Curve. I need to plot the PR curve of each class and the micro and macro average and ...
- 113
1
vote
2
answers
252
views
Why does my ROC curve have a sharp edge?
I was working on a random forest model in R and I got a ROC curve that looks like this. This is very odd since there is no curvature. The data does have mostly qualitative features with only 2-3 ...
- 11
0
votes
0
answers
20
views
Name of the value when sensitivity=specificity
Is there a name for the value along the ROC curve where sensitivity=specificity? This seems like a reasonable way to have a single scalar value to compare classifiers.
- 91
0
votes
0
answers
23
views
Post test probability with a ROC curve
I have data that is normally distributed related to risk of a particular disease. At the median of the distribution, you would expect to observe the population prevalence level of disease P0=0.01. For ...
- 1
0
votes
0
answers
41
views
Compare three different algorithms for anomaly detection
I have 3 different anomaly detection algorithms, that I tested on a mock dataset of 5 elements.
The output of the first and second algorithms, that implement an LSTM, is true/false according to if ...
- 115