Questions tagged [roc]
Receiver Operating Characteristic, also known as the ROC curve.
721
questions
0
votes
0answers
13 views
Panel Data in R - Can I implement a plm regression with categorical data?
I am working (In R) on a panel modelling of an econometric problem.
I have :
one dependent variable : Y with values in real positive numbers
two explanatory variables : VAR_1 (real positive), VAR_2 (...
1
vote
1answer
33 views
ROC for testing goodness of fit
I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression.
I'm a bit confused by the literature where it is mostly just explained as a valid technique to ...
0
votes
0answers
7 views
Is it possible to determine the Probability of Error for the classifier form the ROC curve at a specific point?
Given a classifier with an ROC curve at a specific point, say: (FNR = 0.01, FPR = 0.1).
Is it possible to determine the Probability of Error for the classifier at this point?
0
votes
1answer
18 views
Is it possible for a binary classifier to have lower accuracy, macrof1 and binaryf1 but higher ROC AUC? [duplicate]
I've got the results of two classifiers based on 5 different splits of training and testing sets. Their mean and std of the results are as follow:
Method-------Accuracy -- MacroF1 -- BinaryF1---- ROC ...
1
vote
1answer
22 views
How to generate a ROC curve from ground truth and predictions (in R package pROC)?
I've trained several machine learning models (GLM, SVM, random forest) in R to perform binary classification (predicting the presence of gallstones) and plotted ROC curves/computed AUC using the pROC ...
1
vote
1answer
27 views
How can I better choose my model?
I've made several experimentations in a classification problem, I've got these results:
ROC-AUC Metric:
Train: 0.99, CV: 0.95 +/- 0.01, Test: 0.96
Train: 0.97, CV: 0.94 +/- 0.01, Test: 0.94
Train: 0....
0
votes
0answers
17 views
Treating “probability thresholds” (classification problems) as a hyperparameter
I found this link over here :
https://topepo.github.io/caret/using-your-own-model-in-train.html (section 13.8)
My understanding is that the probability threshold over here is essentially being ...
1
vote
0answers
38 views
Regression model comparison/validation in an independent cohort: other method than c-index
What I have:
I developed a logistic regression model (M1) to predict lymph node metastasis in cancer patients using the variables A + B + C + D + E and a training/validation data set D1, E is the new ...
2
votes
1answer
28 views
ROC Curve for unbounded scores
Say I have a classifier that assigns a score to an image based on whether it has a cat in it. The higher the score, the more likely there's a cat in it. But for this classifier, the value of the score ...
1
vote
0answers
21 views
ROC and AUC for clustering algorithms [closed]
I am working on some clustering algorithms like DBSCAN and local outlier factor. Now i want to know how can I make ROC and AUC curves from clustering results. Do anyone know how can i make RO and AUC ...
0
votes
1answer
14 views
Identifying threshold from Youden Index - Using ROC curve to calculate minimally important change (MIC)
I am trying to determine the minimally important change (MIC) of a frailty instrument using an anchor-based approach outlined below.
Step 1. Fit a logistic regression model between ...
0
votes
1answer
12 views
How to select best models if the ROC AUC score changes drastically at each separate run?
Below are two plots for ROC curves with their AUC mentioned in the legend brackets. How do I shortlist the best models if the scores differ at each run?
Should I rather calculate the ROC AUC only from ...
1
vote
0answers
32 views
Random Forest with train AUC = 1 and test AUC = 58%
I'm trying to understand why my train AUC = 1 while my test AUC is near 58% using random forest.
Context: You are trying to sell a product, and you have historic data about the purchases/noPurchases ...
1
vote
0answers
31 views
What is the name of the probability distribution of ROC-AUC when training machine learning models?
When training ML models like neural networks they are random initialized. That has the effect that the results (ROC-AUC for example) are influenced by random effect. When I train them multiple times ...
7
votes
2answers
138 views
How should two cross-validated logistic regression models be compared?
I'm using 100 times 10-fold repeated cross-validation to assess the ROC-AUC performance improvement of adding a biomarker to an existing model:
Model_A : pred1 + pred2
Model_B :pred1 + pred2 + pred3
I'...
1
vote
0answers
18 views
What do the steps in ROC curve represent? [duplicate]
I have got the following ROC for a binary classification task using scikit-learn's Gaussian Naive Bayes classifier.
The graph shown here shows the ROC curve and has various step like shape. I want to ...
0
votes
0answers
15 views
IntcensROC (R package) - same AUC regardless of input parameters
I have a data frame including various continuous variables (size of a lymph node measured in different ways) and an outcome (disease-free interval).
I have used the intcensROC package to assess the ...
2
votes
0answers
29 views
How can an ROC AUC and its 95%CI be estimated from repeated crossvalidation?
I'm using repeated 100 times 10-fold cross-validation to provide internal validation of a logistic regression model. What is the best way to obtain the ROC AUC and it's 95% CI from this data. If I ...
0
votes
0answers
28 views
ROC Curves for Regression Output
I am working on a broad machine learning-based problem, which can be approached in several different ways. Essentially, my training values are floats between 0.0 and 1.0, and I have approached this in ...
0
votes
0answers
9 views
Help identifying Cumulative Expectation vs Cumulative Identification Graph
In a classification problem, I rank scored observations descending by (calibrated) probability of positive result. If I create a graph that shows cumulative % of total expectation captured on the Y ...
0
votes
1answer
32 views
Is there a way to get the optimal cutoff points based on probability of topic models and the outcomes?
I have topic models probability obtained using LDA topic models method. I’d like to use these probabilities for 5 topics to predict an ...
10
votes
4answers
1k views
Determine how good an AUC is (Area under the Curve of ROC)
I'm currently working on a project involving using different sets of data as a predictor to predict the outcome of out-sample data. I use AUC (Area under the Curve of ROC) to compare the performances ...
1
vote
0answers
55 views
AUC ROC and Varying Thresholds?
I understand that the ROC curve will plot the sensitivity vs FPR for varying thresholds.
For my SVM ML model, I desire a good sensitivity score so I have decreased the threshold to make a positive ...
1
vote
0answers
73 views
Building AUC/ROC curve without probabilities, only with actual/predicted labels
If we dont have access to model and have just actual and predicted labels without probabilities, is it still be possible to plot AUC/ROC curve.
For example can we have the curve from the following ...
0
votes
1answer
35 views
Can I draw the ROC curve from ROC_AUC Score?
The question may look rediculous. The problem in my hand is a multi-class (5 class labels) classification problem and I coded it a year back. I am having the resulsts such as G-mean, F-score, and ...
0
votes
0answers
20 views
How to create an ROC model using three classes
I am trying to create a ROC plot based on response variable with three classes. I believe that is possible based on the answer to this question Here. In the answer provide in this question though, ...
1
vote
0answers
18 views
ROC Curve for data sets with large negative bias
For context, I've read this forum here regarding a similar issue, and it seems the conclusion on there was that precision-recall curves are better-suited for data sets where there is a large negative ...
0
votes
0answers
6 views
Generating a continuous OD graph from a finite set of data points
This is an extract from Donald Bamber's paper regarding OD graphs and ROC plots.
Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristicgraph. ...
1
vote
0answers
26 views
Precision-recall curve for highly imbalanced multi-class classification task
I would like to measure the performance of my network on a multi-class classification problem and wanted to use the precision-recall curve. I have four classes, of which three are extremely ...
1
vote
0answers
21 views
One-sided test for AUC (area under the curve for ROC)
I am trying to determine the power of testing the AUC of a diagnostic test with continuous score against a one-sided alternative. I was thinking I could just use the confidence intervals produced by <...
0
votes
1answer
46 views
Selecting Threshhold from ROC curve [closed]
I plotted an ROC curve for a classification problem and I am looking a way to find out the threshold point for the left most top point - Highest TPR and low FPR. How do I do it in Python ?
1
vote
0answers
21 views
Can you cross validate a regression model using supervised learning?
Our team created a regression model based on a select data size/parameter and achieved a good C-statistic (area under ROC curve).
We faced a few difficulties with submission for publication, so I ...
0
votes
0answers
11 views
Help interpreting my network output
I am having some issues getting my network trained well, and I would be grateful for some feedback. I have a neural network to perform binary classification. The final layer has one node and a sigmoid ...
0
votes
0answers
13 views
Query on Precision-Recall curve
I have an imbalanced dataset - Kaggle's Porto Seguro insurance dataset. I have applied Random Forest and XGBoost classifiers on the imbalanced dataset, under-sampled dataset and over-sampled dataset.
...
1
vote
0answers
49 views
Is the AUC an incoherent measure of classifier performance?
I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ...
0
votes
0answers
31 views
(multiple) Infinite/right censored independent values in diagnostics
We are facing a problem with infinite/right censored independent values.
Case-control study where decay time (mono-exponential function) and half-value time seems to be of great diagnostic accuracy. ...
2
votes
3answers
192 views
How do I plot ROC curves with binary predictions?
Here is a simple dataset of actual and predicted results, with the resulting error matrix. How do I plot an ROC curve with this? I don't understand why the curve is not just four ordered pairs. ...
2
votes
1answer
119 views
Is my AUC too good to be true?
I've been going in circles for months...I want to generate a list of thresholds from my training data so I can see all the thresholds at every sensitivity/1-specificity of a model. You can do this ...
0
votes
0answers
25 views
threshold coded ROC curve interpretaion
I'm trying to figure out ROC curve better and must be kind of dumb but i don't know what are the thresholds of right side of image saying!
On this graph, the y-axis is true positive rate, and the x-...
0
votes
0answers
9 views
How to remove the effect of age on the ROC curve? [duplicate]
I have data set, and the outcome variable is the prognosis (survival or death). I want to evaluate the prognosis of the patient by the SOFA scores, but the problem now is that the age of this data set ...
0
votes
1answer
94 views
Threshold ROC vs. Cut-Off in confusion matrix (binary regression)
I am trying to understand the link between the threshold in ROC-analysis and the threshold defined in classification table. Criterion is binary with 0 or 1.
1) Someone can determine a confusion ...
0
votes
0answers
18 views
Explaining a Conformal Prediction (vs NonConformal)
I'm trying to understand the definition of a conformal prediction and possibly the difference with a non-conformal prediction. What I get as a general idea is that the conformal prediction will be ...
0
votes
0answers
42 views
Cumulative Accuracy Profile(CAP) Curve vs Receiver Operator Characteristic (ROC) Curve
While going through several metrics such as Accuracy and Recall to measure the performance of a Machine Learning model, I found in some articles that ROC Curve and CAP Curve are great for ...
0
votes
0answers
10 views
ROC for a family of repeated experiments
Suppose I have a continuous covariate like age, and I want to use it for RNA expression which means that I have to fit a large number of linear models, one per gene, that predict the expression. ...
0
votes
1answer
37 views
How do I display my output comparing the effect of variables on classification of disease from a random forest analysis in R using an AUROC?
I am using a random forest classification to compare how the classification of disease improves when combining metabolites with two other measures ( visceral fat and CRP-1 levels) to see if adding the ...
2
votes
1answer
51 views
Plot ROC curve in R with the PRROC package
After many hours of research, trial and error and frustration I hope someone here will be able to guide me in the right direction. I am fairly new to R and statistics and can not wrap my hand about ...
1
vote
0answers
24 views
Calculating three average precisions and a single value for ROC from raw predicted class outputs
I'm not a statistician or mathematician so I apologize if I use any terms incorrectly. Please do point out any errors in my use of terminology.
The four values I need are the equivalent of Weka's ROC ...
1
vote
1answer
58 views
Why can't my ROC reach a low PPV?
I am creating a couple of models (RF, SVM, LR) and I want to evaluate all of them on a certain PPV (0.7).
This question and this question helped me write my code:
...
1
vote
1answer
72 views
When is a dataset “too imbalanced” for AUC ROC and PR is preferred?
I’ve read that precision-recall (PR) curves are preferred over AUC-ROC curves when a dataset is imbalanced as there’s more of a focus on the model’s performance in correctly identifying the minority/...
0
votes
1answer
22 views
how to increase the output values without increasing datasets?
I am using MatLab 2017a. I have dataset of 12 video. I have to plot ROC curve of this dataset. For this purpose, I have used perfcurve command.
...
