Questions tagged [roc]

Receiver Operating Characteristic, also known as the ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system

Filter by
Sorted by
Tagged with
1 vote
1 answer
14 views

Selecting best classification probability threshold with ROC/AUC doesn't necessarily improve F1 score

I read that probability based binary classifiers have 0.5 as default probability threshold for getting hard 0/1 labels (in scikit-learn for example) but this could be fine-tuned with methods like ...
  • 715
2 votes
2 answers
82 views

Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?

$AUC=0.5$ and an ROC curve of a 45-degree line often are considered the baseline performance of a model, one that gets absolutely nothing from the features. If we predict the same (prior) probability ...
  • 35.8k
0 votes
0 answers
9 views

how to get the diagnostic power of more than one variable combined?

I have three antibody measurements to predict a condition. I am doing ROC analysis for each of the antibodies to check what is the diagnostic power for each one. But I want to try what is the ...
  • 841
1 vote
0 answers
11 views

Does a model with 0.5 AUROC imply an average precision equal to the proportion of positive examples?

A random model has an area under the ROC curve equal to 0.5. We also know that a random model has an area under the Precision-Recall curve equal to the proportion (p) of positive examples. Then, here'...
  • 111
0 votes
0 answers
11 views

Calculating AUC from Match Score Matrix

I am reading "Introduction to Biometrics" (2011) by Jain, Ross, and Nandakumar. They propose the following scenario: Consider a scenario where the biometric data (e.g., right index ...
1 vote
1 answer
30 views

AUC-ROC interpretation

I performed AUC-ROC curve analysis to confirm the biomarker potential of genes that were significant in gene expression results. This is how the graphics came out in the genes I determined. How can I ...
  • 11
0 votes
1 answer
23 views

ROC curve illustrates a perfect classificator however other metrics show worse values

how is it to explain that ROC illustrates such a perfect classificator however other metrics represent something different? The evalaution was done on CV of 5. ...
  • 196
0 votes
1 answer
29 views

Is there something equivalent to a ROC curve for logistic GLMMs? [closed]

When I read previously about logistic regression, I recall some talk about ROC curves and how they can be good metrics for the sensitivity/specificity of a logistic regression as well as a goodness of ...
10 votes
2 answers
780 views

Is higher AUC always better?

Let's say we measure binary classifier performance by ROC graph, and we have two separate models with distinct AUC (The Area Under the Curve) values. Is the model with the higher AUC value always ...
  • 317
0 votes
0 answers
23 views

Is a Sensitivity-Specificity curve equal to a horizontally flipped ROC?

So I need to plot a Sensitivity-Specificity curve. Since ROCs represent TPR (sensitivity) against FPR (or 1 - Specificity), can I just plot TPR against 1 - FPR as in the code below to obtain a Sens-...
1 vote
0 answers
34 views

ROC curve - Question with exercise

I'm studying the ROC curve theory but I'm struggling with an apparently simple exercise. To recap what I know: "The ROC curve is a graphical plot that illustrates the diagnostic ability of a ...
1 vote
0 answers
37 views

Can't get AUC higher than 0.62 using XGBoost or Random forest [duplicate]

I have a dataframe with multiple numeric predictive features, and one binary target feature. 63% of the data belongs to class 0, and the rest to class ...
  • 153
0 votes
0 answers
17 views

does the ROC curve of a committee based predictor have any meaning?

would appreciate it if you'd take a moment to read the pipeline I've described below - it relates to how a learner that is based on a committee should be optimized w.r.t the threshold of the ROC curve....
  • 115
0 votes
0 answers
23 views

Evaluating country-level indicators through ROC analysis

Good afternoon. I have defined a binary classification problem. Classes are based on country-level data on child undernourishment whereby a positive class is assigned to a country whenever it ...
1 vote
0 answers
14 views

Interpreting the fitted model outputs from a covariate-adjusted ROC curve in the AROC package

Below I have fitted a covariate-adjusted ROC curve (AROC) using a semiparametric Bayesian normal linear regression model available in the R package “AROC”. However, I am not clear on how to interpret/...
1 vote
0 answers
23 views

Computing a Weighted AUC Metric

Consider the standard ROC-AUC metric, which has been shown to be equivalent to the normalized Wilcoxon-Mann-Whitney: $$ AUC = \frac{1}{N^{(+)}N^{(-)}} \sum_{i}\sum_{j}1[\hat{p}_i^{(+)} > \hat{p}_j^{...
  • 281
0 votes
2 answers
53 views

Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset

I am currently doing a logistic regression analysis and wanted to cross validate the results. Now that I have the probabillities calculated for each fold, I made a ROC Curve for each one as well. But ...
  • 43
1 vote
2 answers
45 views

Exchange the x-axis and y-axis in order to flip ROC curve

I have a continuous variable X and a binary response outcome D. The ROC curve I got for D with respect to X is below the diagonal line, as shown in the following picture. In this ROC curve, the true ...
  • 11
1 vote
0 answers
17 views

consistent gap between training and validation metrics

I am training a neural network (Deep and cross network) for a multi-label classification task (~700 labels). I am seeing a uniform gap between training and validation results on various metrics. E.g. ...
  • 311
1 vote
2 answers
30 views

Using a ROC Plot to interpret specific scores

I have a binary classifier which outputs a given score to differentiate normal (low score) from abnormal (high score) cases. The score itself however is non-interpretable to others. I know a ROC plot ...
0 votes
2 answers
22 views

Individual Measurement in Overlapping Distributions

I have a situation where I have measurements taken from samples of healthy and pathological populations. The pathological populations tend to have much higher measurements, but the distributions are ...
0 votes
0 answers
32 views

ROC curve representation in paper

Recently I crossed a paper, which authors presented their ROC curve results in the following way: 44,7,0,19,18,74,15,8,27,24,2,0,0,0,2,0,0,0,0,0,0,0, 0,0,0,183,42,36,46,59,25,6,2,2,15,330,81,59,86,1 ...
  • 405
4 votes
1 answer
254 views

how to compare the AUCs of ROC from nested models and non-nested models from published papers

I'm interested in comparing two AUCs from ROC in two nested models and two AUCs in non-nested models but using the same dataset in a paper. I know DeLong test can be used to compare them, but the ...
  • 143
0 votes
0 answers
18 views

Constant model AUC

In sklearn, if one uses a model that makes constant predictions (all predictions equal to a given value $x$), the ROC AUC value for that model is 0.5. However, I don't see that clearly from the area ...
1 vote
0 answers
47 views

Ties in AUC probabilistic definition

According to the ROC AUC definition in here: The correct statement is that ROC AUC is the probability a randomly-chosen positive example is ranked more highly than a randomly-chosen negative example. ...
1 vote
0 answers
36 views

How to interpret ROC curve if my FPR (X-axis) values ranges from 0 to 0.02?

I am plotting the ROC curve for my classification problem. The results I am getting for the problem are TPR ranges from 0 to 1, but the FPR ranges from 0 to 0.02. I have plotted the ROC curve by ...
2 votes
1 answer
31 views

Examples of problems where ROC curves are more useful than PR curves

I understand the formal differences between a ROC curve and a PR curve, but I am trying to build more intuition. For common applications such as Search and disease prediction, PR curves are more ...
  • 995
1 vote
1 answer
25 views

Can the AUC of the ROC be interpreted as the average recall?

The PR curve plots the precision as a function of the recall, and the AUC can be interpreted as the average precision. The ROC plots the recall as a function of the specificity. Can we interpret the ...
  • 995
0 votes
1 answer
25 views

When should I balance my data using AUROC and AUPRC?

I want to report the AUROC and the AUPRC of a prediction model using an unbalanced dataset. Is it correct that I have to balance my data to calculate the AUROC but leave the data unbalanced to ...
  • 21
1 vote
0 answers
14 views

AUROC or AUPRC? [duplicate]

I have a highly imbalanced dataset and want to evaluate the performance of a score. Is it better to balance the data and calculate the area under the receiver operating characteristic or to use the ...
  • 21
0 votes
0 answers
77 views

Confusion matrix vs ROC-AUC curve

I'm using the lares packages in combination with H2O and with ...
  • 2,441
2 votes
2 answers
62 views

How is AUC helpful when we only need one threshold of a classifier

AUC is a summation of performance at different thresholds, but do we only care about a good performance at one threshold? Imagine a classifier with a low ROC but shots up at point of a low FP and high ...
  • 21
0 votes
0 answers
13 views

Area Under Precision-Recall and Area Under ROC curve for different amount of observations

I am doing a research and thus comparing some algorithms for binary classification. Worth to mention that, the data set is highly imbalanced i.e., the minority class is only 0.2%. Notation: Area Under ...
  • 1
0 votes
0 answers
17 views

Understand AUCs of different models

I'm testing two models against each other: provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %. provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
  • 2,441
0 votes
0 answers
22 views

what is the acceptable value for acceptable cut point while minimizes distance between ROC plot and point (0,1)

If we would like to find the optimal cut point using minimizes distance between ROC plot and point (0,1), is it a acceptable value in the field for the roc01 value? I have searched for the reference ...
  • 1
0 votes
0 answers
32 views

What is my best option to evaluate a predictive (or proxy) variable?

I have a list of thunder flashes, and I'm trying to find if a meteorological variable (CAPE) is a good predictable variable (or proxy) for theses flashes. In my thoughts, I want to evaluate this proxy ...
0 votes
0 answers
23 views

Make ROC graph with my problematic [duplicate]

For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic. I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
0 votes
0 answers
45 views

Determining cut-off (Youden) without causing data leakage

I´m new to coding and machine learning so this might sound like a stupid question. I´m using Logistic regression, RF and SVM to model corporate defaults in R. However i´m a bit concerned of what could ...
  • 1
0 votes
1 answer
31 views

How to interpret this confusion matrix and roc curve?

I got these two images for evaluating a RF: I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %? By looking only at ...
  • 2,441
2 votes
1 answer
78 views

AUC - Logistic Regression versus LDA, and Naive Bayes

everyone! I am a newbie on machine learning, and I am now interested on classification modeling. I used logistic regression, linear discriminant analysis (LDA), and naive Bayes on my notebook DataCamp ...
0 votes
0 answers
9 views

Is there any Statistic method which could reflect the diagnose value when the prevalence of special type of characteristic is lower

Our present study is conducting a novel diagnosis test for predicting a disease by the X-ray. The gold-standard is a minimally invasive procedure can test a tissue sample for the disease. In our ...
1 vote
1 answer
228 views

Logistic Regression: What is the value for precision when recall (true positive rate) is 0?

A quick overview of definitions before I get into the question: True Positive (TP): An actual positive that the model classified as positive False Positive (FP): An actual negative that the model ...
  • 211
1 vote
1 answer
54 views

Is there a good reason for using AUROC on imbalanced dataset?

So I just learned about AUROC. When I read this thread, it seems like AUROC is not a great metric for imbalanced dataset. One answer even says it shouldn't be used to compare models. However, I am ...
  • 135
3 votes
2 answers
351 views

Difference between ROC-AUC and Multiclass AUC (MAUC)

I am trying to understand the interpretation of these metrics in a multiclass scenario: ROC-AUC and MAUC. Scikit-learn provides ...
  • 231
0 votes
0 answers
79 views

My data can be approximated with normal distribution (multimodal). How can I find the reasons and explain this behaviour?

I use DeLonge method to compare two ROC AUCS. The result of it is Z-score. Both ROC AUCs obtained from LDA (linear discriminant analysis) from sklearn package. The ...
0 votes
0 answers
255 views

Precision-Recall curve and AUC in multi-class problems using R

I'm trying to evaluate a RandomForest model for multi-class classification using the Area Under Precision-Recall Curve. I need to plot the PR curve of each class and the micro and macro average and ...
  • 113
1 vote
2 answers
252 views

Why does my ROC curve have a sharp edge?

I was working on a random forest model in R and I got a ROC curve that looks like this. This is very odd since there is no curvature. The data does have mostly qualitative features with only 2-3 ...
  • 11
0 votes
0 answers
20 views

Name of the value when sensitivity=specificity

Is there a name for the value along the ROC curve where sensitivity=specificity? This seems like a reasonable way to have a single scalar value to compare classifiers.
0 votes
0 answers
23 views

Post test probability with a ROC curve

I have data that is normally distributed related to risk of a particular disease. At the median of the distribution, you would expect to observe the population prevalence level of disease P0=0.01. For ...
0 votes
0 answers
41 views

Compare three different algorithms for anomaly detection

I have 3 different anomaly detection algorithms, that I tested on a mock dataset of 5 elements. The output of the first and second algorithms, that implement an LSTM, is true/false according to if ...
  • 115

1
2 3 4 5
17