Questions tagged [roc]
Receiver Operating Characteristic, also known as the ROC curve.
748
questions
1
vote
1answer
14 views
Interpreting classification metrics for multiclass imbalance
I am at the point of reporting my results in a research article conducted. The dataset is highly imbalanced with class 1 and ...
2
votes
1answer
39 views
Why do we look at multiple thresholds in AUC
For AUC ROC, why do we have a graph of multiple thresholds when in the end, we will only use one of those thresholds (so why not just choose the threshold and compare that one value across models)?
0
votes
0answers
14 views
Micro average for AUC/ROC
The yellowbrick documentation has an example of AUC/ROC here. It seems odd to me the micro average would have an AUC much bigger than the individual classes or macro average. Is there some reason the ...
0
votes
0answers
12 views
Why is ROC a ranking metric and how do we understand it as a c-statistic?
I have been stuck on this topic for quite a long time. I still do not understand why AUROC is a ranking metric. In particular, what does "ranking" mean in AUROC? When I tried to code ROC out ...
0
votes
0answers
28 views
Why K-fold cross validation of ROC curve is meaningful in research? [closed]
I saw that many people draw multiple ROC curves in the same plot to compare models in medical imaging.
However, I was wondering why that would be a meaningful work to draw cross validation ROC curve ...
0
votes
0answers
11 views
Can you do hyperparameter tuning using a PR curve? Moreover, would this still be considered a “PR curve”?
I am creating a graphical representation of PR results across various hyperparameter changes in an imbalanced dataset (model used was an SVM). I'm wondering if one would still consider this a "PR ...
0
votes
0answers
11 views
Why does using probabilities result in higher ROC_AUC scores?
in the most recent playground competition on kaggle (https://www.kaggle.com/c/tabular-playground-series-mar-2021/overview/evaluation) we once again have an evaluation via the area under the roc curve.
...
0
votes
1answer
14 views
imbalanced classes: ROC_AUC vs Precision_Recall AUC
I am dealing with a highly imbalanced classes problem. Accuracy is of course not a good performance metric in such cases, So I want to calculate either ROC AUC sore ...
2
votes
1answer
42 views
Why is the diagonal line in ROC space called the random guess?
all. I'm a student who studies machine learning.
When I found the wiki over the ROC(Reciever Operation Characteristic), I had one question.
According to the wiki, ROC space has a diagonal line and ...
1
vote
0answers
18 views
ROC AUC for multiclass problem
Just some quick questions to clarify my doubt please.
I know that one can get precision/recall for each class in a multiclass problem, e.g. in this classification ...
0
votes
0answers
22 views
Interpretation of Youden's J Statistic
Youden's J statistic is defined as J = sensitivity + specificity - 1, and is equal to the vertical distance between the chance line and the ROC curve for a classifier. I'm having trouble describing ...
1
vote
0answers
26 views
ROC curve as predictive tool in human performance--relationship between $\beta$, $X_C$, and signal probability
I am taking an engineering psychology course in which we are using ROC curves to evaluate human performance in signal detection. Many of the questions on this site about ROC curves are using them as a ...
0
votes
0answers
8 views
One-vs-All ROC curve computation
Suppose you have a classification model which can predict any one of N classes. The model can also output the prior probabilities of that prediction, one for each class. These probabilities can be ...
0
votes
0answers
17 views
Optimal cut-off value for continuous covariate with binary outcome [duplicate]
Research question: What is the optimal cut-off value for a continuous covariate (age) with a binary outcome (suitable or not for a certain treatment). I realize that we lose information by ...
1
vote
0answers
25 views
Inconsistent ROC-AUC
I'm training VGAE to predict edges and I'm getting a good loss (BCEwithLogits) values but the ROC-AUC seems bit inconsistent and not very progressive. I'm not sure how to diagnose it.
The model is ...
0
votes
0answers
11 views
Adaptive cutpoint selection on ROC curves based on changing environments
I have built a classic binary classifier and constructed a ROC curve for it, like the following:
In this case, the positive class represents "bad" things that should be excluded. In the ...
0
votes
1answer
30 views
Why is the cutoff for AUC (area under ROC curve) to be considered an acceptable classfier at .7 and what is it acceptable for? [duplicate]
From reading about interpreting the area under a ROC curve, it seems the general consensus is that anything from .7 - .8 is considered an acceptable classifier. What is it considered "acceptable&...
0
votes
0answers
17 views
For imbalanced datasets, is it necessary to use undersampling or upsampling if one evaluates the performance of ML classifier using PR curves?
Previously, I would have assumed evaluating the classification performance of Decision tree and SVM with a PR curve would obviate the need for under/over-sampling since it doesn't evaluate the true ...
0
votes
1answer
27 views
Comparing the discriminative ability of two comorbidity indices for mortality: should I go for ROC and AUC?
Different comorbidity indices are widely used tools for describing patients' comorbidity statuses. A large score = many illnesses, and vice versa. I would like to compare two such indices to examine ...
0
votes
0answers
10 views
ROC curve with multiple thresholds / equivalent rank measure
I have a model that has X features, and I can create ROC curves based on the residuals for each feature (so that each feature has its own curve, that determines how good that feature's residuals are ...
0
votes
0answers
11 views
ROC convex hull - realisable classifiers
In the paper Realisable Classifiers Theorem 1 shows that there exists a realisable classifier r_i which lies on a line L_ab ...
0
votes
0answers
8 views
How to calculate labels and scores for a feature detector without classification or segmentation?
I am using SIFT feature detector to detect features from ground truth image and test image. I am using the location and scale of ground truth feature to define the predicted features on test image. I ...
0
votes
1answer
49 views
Is it possible to get low AUC score but high Precision and Recall?
I am doing classification on a fairly imbalanced dataset (about 1:2 ratio). I have so far so far tried lasso and logistic regression. I didn't downsample the dataset because the sample size is low (...
2
votes
1answer
37 views
Should I use predict_proba or predict when computing metrics
I need to compute some metrics for binary classification. I see that many times some people use the probability:
...
0
votes
0answers
14 views
Bias of False Positive Rate (FPR) estimator
When evaluating a classifier's false positive rate on a randomly sampled test set (of size much smaller than the population), is an estimator for the false positive rate considered a "ratio ...
0
votes
0answers
11 views
roc auc for small class imbalance
I have a classification problem with class imbalance(1:6). I'd like to know if roc_auc is a valid metric for this level of imbalance. I know it's not good for severe imbalance, but what about a case ...
1
vote
1answer
26 views
3 models which I have built result in the exact same result, how can it be? [closed]
I have 3 models which I have built,
I have used the following libraries:
...
1
vote
2answers
59 views
Showing that $P(X_1>X_2) = \int_{0}^1 P(X_1>X_2 | X_2=x) f_{X_2}(x) dx$
I am going through this post in trying to prove the probabilistic interpretation of the AUC for a ROC Curve (for a classifier):
The AUC for a ROC curve is the the probability of the classifier ...
0
votes
1answer
49 views
What does it mean if ROC curves (training ROCs) are very smooth?
I am a beginner to statistical / machine learning modelling and have a beginner question.
What does it mean if ROC curves are very smooth? That is they actually look like curves.
In textbooks, these ...
0
votes
0answers
45 views
Is KS statstic (in classification) from ROC Curve same as K-S from cum Gain curve or PR Curve?
This is related to classification problems (specifically binary classification problems covered in scoring) where curves referenced below are used. I understand each of these curves for a classifier ...
0
votes
1answer
26 views
How is the cost weight function $w_G(c)$ implicitly used by the AUC calculated?
In this article by David Hand an implicit function of the classification cost ratio is calculated for a specific dataset, resulting in a discrete distribution:
This is defined as
$$ w_G(c) = \pi_0 ...
0
votes
0answers
48 views
Calibrating probability thresholds based on ROC curve for multiclass classification
I have built a network for the classification of three classes. The network consists of a CNN followed by two fully-connected layers. The CNN consists of convolutional layers, followed by batch ...
6
votes
1answer
301 views
When are ROC curves to compare imaging tests valid? (Focus on the example below)
I would like to ask what criticism could be raised in the following case example: In this paper they test a way of detecting narrowing of the cervical canal on radiographs using a ratio of ...
0
votes
0answers
26 views
test how good a threshold is?
I have developed a threshold and wondering how can I test that threshold to see how good it is? I tried precision, recall, and F1 score metrics, and the results were promising. However, I'm wondering ...
0
votes
0answers
48 views
1
vote
1answer
28 views
The calculation results of coords and ci.coords in the pROC package are inconsistent. Which one should I choose?
I have a set of data. When calculating with the pROC package, the sensitivity results obtained by coords and ...
1
vote
0answers
77 views
How do I evaluate the likelihood of the binormal smoothed ROC curve?
As I understand the binormal model for ROC curves assumes that the decision variable can be monotonically transformed so that both the case and control values are normally distributed. Under this ...
6
votes
1answer
268 views
F1 score, PR or ROC curve for regression
Due to my background as a pure biologist, I've been struggling with the comment acquired from a reviewer about the accuracy test used in my regression study. While I stick to MSE, MAE and R2 as the ...
0
votes
0answers
20 views
I compared means of var X for two groups (t test). How calculate ROC for the classifier on these two groups based on variable X?
How to calculate roc curve after performing t test (in sas or in R)? I compared means of variable X for two groups using t test. P-value is <0.05. How can I also calculate ROC curve for the ...
0
votes
0answers
15 views
2
votes
0answers
21 views
Why does the Youden rule does not recommend a threshold of 0.5 on balanced data?
Suppose I have a logistic regression model estimated using a balanced target (equal group sizes). My questions concern the optimal threshold for prediction and it's relationship with the Youden's rule ...
0
votes
0answers
15 views
is there a name for the generalization of ROC AUC to ordinal data with more than 2 classes? “probability of correct ordering of a pair”
https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve
the area under the ROC curve is the probability of a random positive and a random negative being scored / ranked in ...
0
votes
0answers
81 views
Equal Error Rate (EER) Metric - Why is lower better?
With the Equal Error Rate (EER) Metric, why is the lower the value the better (more accurate binary classifier system)?
Also, when finding the EER on the ROC curve, is it correct in saying the EER is ...
2
votes
1answer
48 views
How to test significance of two ROC with MLeval
I've two ROC derived from caret and I'd like to test if the relative curves are statistically different:
...
3
votes
0answers
69 views
Is an off diagonal ROC curve not always better than random?
I'm trying to better understand the ROC when used for ML model classification and was looking at this explaining curve, explaining what is better and worse. However, I am thinking, contrary what is ...
0
votes
0answers
44 views
Panel Data in R - Can I implement a plm regression with categorical data?
I am working (In R) on a panel modelling of an econometric problem.
I have :
one dependent variable : Y with values in real positive numbers
two explanatory variables : VAR_1 (real positive), VAR_2 (...
1
vote
1answer
113 views
ROC for testing goodness of fit
I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression.
I'm a bit confused by the literature where it is mostly just explained as a valid technique to ...
0
votes
0answers
11 views
Is it possible to determine the Probability of Error for the classifier form the ROC curve at a specific point?
Given a classifier with an ROC curve at a specific point, say: (FNR = 0.01, FPR = 0.1).
Is it possible to determine the Probability of Error for the classifier at this point?
0
votes
1answer
26 views
Is it possible for a binary classifier to have lower accuracy, macrof1 and binaryf1 but higher ROC AUC? [duplicate]
I've got the results of two classifiers based on 5 different splits of training and testing sets. Their mean and std of the results are as follow:
Method-------Accuracy -- MacroF1 -- BinaryF1---- ROC ...
1
vote
1answer
380 views
How to generate a ROC curve from ground truth and predictions (in R package pROC)?
I've trained several machine learning models (GLM, SVM, random forest) in R to perform binary classification (predicting the presence of gallstones) and plotted ROC curves/computed AUC using the pROC ...
