Questions tagged [roc]
Receiver Operating Characteristic, also known as the ROC curve.
735
questions
0
votes
0answers
14 views
Bias of False Positive Rate (FPR) estimator
When evaluating a classifier's false positive rate on a randomly sampled test set (of size much smaller than the population), is an estimator for the false positive rate considered a "ratio ...
0
votes
0answers
10 views
roc auc for small class imbalance
I have a classification problem with class imbalance(1:6). I'd like to know if roc_auc is a valid metric for this level of imbalance. I know it's not good for severe imbalance, but what about a case ...
1
vote
1answer
24 views
3 models which I have built result in the exact same result, how can it be? [closed]
I have 3 models which I have built,
I have used the following libraries:
...
1
vote
2answers
50 views
Showing that $P(X_1>X_2) = \int_{0}^1 P(X_1>X_2 | X_2=x) f_{X_2}(x) dx$
I am going through this post in trying to prove the probabilistic interpretation of the AUC for a ROC Curve (for a classifier):
The AUC for a ROC curve is the the probability of the classifier ...
0
votes
1answer
45 views
What does it mean if ROC curves (training ROCs) are very smooth?
I am a beginner to statistical / machine learning modelling and have a beginner question.
What does it mean if ROC curves are very smooth? That is they actually look like curves.
In textbooks, these ...
0
votes
0answers
8 views
Is KS statstic (in classification) from ROC Curve same as K-S from cum Gain curve or PR Curve?
This is related to classification problems (specifically binary classification problems covered in scoring) where curves referenced below are used. I understand each of these curves for a classifier ...
0
votes
1answer
23 views
How is the cost weight function $w_G(c)$ implicitly used by the AUC calculated?
In this article by David Hand an implicit function of the classification cost ratio is calculated for a specific dataset, resulting in a discrete distribution:
This is defined as
$$ w_G(c) = \pi_0 ...
0
votes
0answers
23 views
Calibrating probability thresholds based on ROC curve for multiclass classification
I have built a network for the classification of three classes. The network consists of a CNN followed by two fully-connected layers. The CNN consists of convolutional layers, followed by batch ...
6
votes
1answer
296 views
When are ROC curves to compare imaging tests valid? (Focus on the example below)
I would like to ask what criticism could be raised in the following case example: In this paper they test a way of detecting narrowing of the cervical canal on radiographs using a ratio of ...
0
votes
0answers
23 views
test how good a threshold is?
I have developed a threshold and wondering how can I test that threshold to see how good it is? I tried precision, recall, and F1 score metrics, and the results were promising. However, I'm wondering ...
0
votes
0answers
40 views
1
vote
1answer
19 views
The calculation results of coords and ci.coords in the pROC package are inconsistent. Which one should I choose?
I have a set of data. When calculating with the pROC package, the sensitivity results obtained by coords and ...
1
vote
0answers
75 views
How do I evaluate the likelihood of the binormal smoothed ROC curve?
As I understand the binormal model for ROC curves assumes that the decision variable can be monotonically transformed so that both the case and control values are normally distributed. Under this ...
6
votes
1answer
220 views
F1 score, PR or ROC curve for regression
Due to my background as a pure biologist, I've been struggling with the comment acquired from a reviewer about the accuracy test used in my regression study. While I stick to MSE, MAE and R2 as the ...
0
votes
0answers
19 views
I compared means of var X for two groups (t test). How calculate ROC for the classifier on these two groups based on variable X?
How to calculate roc curve after performing t test (in sas or in R)? I compared means of variable X for two groups using t test. P-value is <0.05. How can I also calculate ROC curve for the ...
0
votes
0answers
13 views
2
votes
0answers
20 views
Why does the Youden rule does not recommend a threshold of 0.5 on balanced data?
Suppose I have a logistic regression model estimated using a balanced target (equal group sizes). My questions concern the optimal threshold for prediction and it's relationship with the Youden's rule ...
0
votes
0answers
11 views
is there a name for the generalization of ROC AUC to ordinal data with more than 2 classes? “probability of correct ordering of a pair”
https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve
the area under the ROC curve is the probability of a random positive and a random negative being scored / ranked in ...
0
votes
0answers
40 views
Equal Error Rate (EER) Metric - Why is lower better?
With the Equal Error Rate (EER) Metric, why is the lower the value the better (more accurate binary classifier system)?
Also, when finding the EER on the ROC curve, is it correct in saying the EER is ...
1
vote
1answer
38 views
How to test significance of two ROC with MLeval
I've two ROC derived from caret and I'd like to test if the relative curves are statistically different:
...
3
votes
0answers
57 views
Is an off diagonal ROC curve not always better than random?
I'm trying to better understand the ROC when used for ML model classification and was looking at this explaining curve, explaining what is better and worse. However, I am thinking, contrary what is ...
0
votes
0answers
33 views
Panel Data in R - Can I implement a plm regression with categorical data?
I am working (In R) on a panel modelling of an econometric problem.
I have :
one dependent variable : Y with values in real positive numbers
two explanatory variables : VAR_1 (real positive), VAR_2 (...
1
vote
1answer
66 views
ROC for testing goodness of fit
I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression.
I'm a bit confused by the literature where it is mostly just explained as a valid technique to ...
0
votes
0answers
10 views
Is it possible to determine the Probability of Error for the classifier form the ROC curve at a specific point?
Given a classifier with an ROC curve at a specific point, say: (FNR = 0.01, FPR = 0.1).
Is it possible to determine the Probability of Error for the classifier at this point?
0
votes
1answer
22 views
Is it possible for a binary classifier to have lower accuracy, macrof1 and binaryf1 but higher ROC AUC? [duplicate]
I've got the results of two classifiers based on 5 different splits of training and testing sets. Their mean and std of the results are as follow:
Method-------Accuracy -- MacroF1 -- BinaryF1---- ROC ...
1
vote
1answer
146 views
How to generate a ROC curve from ground truth and predictions (in R package pROC)?
I've trained several machine learning models (GLM, SVM, random forest) in R to perform binary classification (predicting the presence of gallstones) and plotted ROC curves/computed AUC using the pROC ...
1
vote
1answer
30 views
How can I better choose my model?
I've made several experimentations in a classification problem, I've got these results:
ROC-AUC Metric:
Train: 0.99, CV: 0.95 +/- 0.01, Test: 0.96
Train: 0.97, CV: 0.94 +/- 0.01, Test: 0.94
Train: 0....
0
votes
0answers
50 views
Treating “probability thresholds” (classification problems) as a hyperparameter
I found this link over here :
https://topepo.github.io/caret/using-your-own-model-in-train.html (section 13.8)
My understanding is that the probability threshold over here is essentially being ...
1
vote
0answers
41 views
Regression model comparison/validation in an independent cohort: other method than c-index
What I have:
I developed a logistic regression model (M1) to predict lymph node metastasis in cancer patients using the variables A + B + C + D + E and a training/validation data set D1, E is the new ...
3
votes
1answer
31 views
ROC Curve for unbounded scores
Say I have a classifier that assigns a score to an image based on whether it has a cat in it. The higher the score, the more likely there's a cat in it. But for this classifier, the value of the score ...
1
vote
0answers
24 views
ROC and AUC for clustering algorithms [closed]
I am working on some clustering algorithms like DBSCAN and local outlier factor. Now i want to know how can I make ROC and AUC curves from clustering results. Do anyone know how can i make RO and AUC ...
0
votes
1answer
39 views
Identifying threshold from Youden Index - Using ROC curve to calculate minimally important change (MIC)
I am trying to determine the minimally important change (MIC) of a frailty instrument using an anchor-based approach outlined below.
Step 1. Fit a logistic regression model between ...
0
votes
1answer
23 views
How to select best models if the ROC AUC score changes drastically at each separate run?
Below are two plots for ROC curves with their AUC mentioned in the legend brackets. How do I shortlist the best models if the scores differ at each run?
Should I rather calculate the ROC AUC only from ...
1
vote
0answers
42 views
Random Forest with train AUC = 1 and test AUC = 58%
I'm trying to understand why my train AUC = 1 while my test AUC is near 58% using random forest.
Context: You are trying to sell a product, and you have historic data about the purchases/noPurchases ...
1
vote
0answers
37 views
What is the name of the probability distribution of ROC-AUC when training machine learning models?
When training ML models like neural networks they are random initialized. That has the effect that the results (ROC-AUC for example) are influenced by random effect. When I train them multiple times ...
7
votes
2answers
174 views
How should two cross-validated logistic regression models be compared?
I'm using 100 times 10-fold repeated cross-validation to assess the ROC-AUC performance improvement of adding a biomarker to an existing model:
Model_A : pred1 + pred2
Model_B :pred1 + pred2 + pred3
I'...
1
vote
0answers
19 views
What do the steps in ROC curve represent? [duplicate]
I have got the following ROC for a binary classification task using scikit-learn's Gaussian Naive Bayes classifier.
The graph shown here shows the ROC curve and has various step like shape. I want to ...
0
votes
0answers
15 views
IntcensROC (R package) - same AUC regardless of input parameters
I have a data frame including various continuous variables (size of a lymph node measured in different ways) and an outcome (disease-free interval).
I have used the intcensROC package to assess the ...
2
votes
0answers
33 views
How can an ROC AUC and its 95%CI be estimated from repeated crossvalidation?
I'm using repeated 100 times 10-fold cross-validation to provide internal validation of a logistic regression model. What is the best way to obtain the ROC AUC and it's 95% CI from this data. If I ...
0
votes
0answers
28 views
ROC Curves for Regression Output
I am working on a broad machine learning-based problem, which can be approached in several different ways. Essentially, my training values are floats between 0.0 and 1.0, and I have approached this in ...
0
votes
1answer
19 views
Help identifying Cumulative Expectation vs. Cumulative Identification Graph
In a classification problem, I rank scored observations descending by (calibrated) probability of positive result. If I create a graph that shows cumulative % of total expectation captured on the Y-...
0
votes
1answer
49 views
Is there a way to get the optimal cutoff points based on probability of topic models and the outcomes?
I have topic models probability obtained using LDA topic models method. I’d like to use these probabilities for 5 topics to predict an ...
10
votes
4answers
1k views
Determine how good an AUC is (Area under the Curve of ROC)
I'm currently working on a project involving using different sets of data as a predictor to predict the outcome of out-sample data. I use AUC (Area under the Curve of ROC) to compare the performances ...
1
vote
0answers
126 views
AUC ROC and Varying Thresholds?
I understand that the ROC curve will plot the sensitivity vs FPR for varying thresholds.
For my SVM ML model, I desire a good sensitivity score so I have decreased the threshold to make a positive ...
1
vote
0answers
164 views
Building AUC/ROC curve without probabilities, only with actual/predicted labels
If we dont have access to model and have just actual and predicted labels without probabilities, is it still be possible to plot AUC/ROC curve.
For example can we have the curve from the following ...
0
votes
1answer
38 views
Can I draw the ROC curve from ROC_AUC Score?
The question may look rediculous. The problem in my hand is a multi-class (5 class labels) classification problem and I coded it a year back. I am having the resulsts such as G-mean, F-score, and ...
0
votes
0answers
21 views
How to create an ROC model using three classes
I am trying to create a ROC plot based on response variable with three classes. I believe that is possible based on the answer to this question Here. In the answer provide in this question though, ...
1
vote
0answers
21 views
ROC Curve for data sets with large negative bias
For context, I've read this forum here regarding a similar issue, and it seems the conclusion on there was that precision-recall curves are better-suited for data sets where there is a large negative ...
0
votes
0answers
9 views
Generating a continuous OD graph from a finite set of data points
This is an extract from Donald Bamber's paper regarding OD graphs and ROC plots.
Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristicgraph. ...
1
vote
0answers
32 views
Precision-recall curve for highly imbalanced multi-class classification task
I would like to measure the performance of my network on a multi-class classification problem and wanted to use the precision-recall curve. I have four classes, of which three are extremely ...
