Questions tagged [roc]
Receiver Operating Characteristic, also known as the ROC curve.
629
questions
0
votes
0answers
23 views
Finding optimal cutoff point [duplicate]
.Hello,everyone.
I am studying the influence of one biomarker on multiple disease characteristics, and I would like to calculate its cutoff point. I created univariate ROC curves to investigate the ...
0
votes
1answer
12 views
Possible reasons for AUC=1 (from a fitted glm model)?
I am running high-throughput microarray data (methylation array), and after running univariate, lasso and cross-validation lasso analyses, I was able to come down to a list of 15 probes (predictors).
...
1
vote
1answer
41 views
ROC Curve Horizontal Line
I'm trying to conceptually understand 'why' the horizontal gray line in ROC curve below would be considered random.
I know the basics of the ROC in that the farther the curved black line is from the ...
0
votes
0answers
10 views
Determining Boundary between True Positive/Negative Results
I have a large dataset of distributions similar to the histogram below. Each distribution is a part of an independent set that's being assessed one-by-one, and I'm looking to automate the process.
...
0
votes
1answer
27 views
Calculate AUC using predicted values and labels from a 5 fold classification?
I have a classifier for a binary problem. That has outputs between 0 and 1 for predictions for the two class A or B (for example sunny, not sunny). The classifier has ran on 5 unique folds of the data ...
0
votes
0answers
18 views
How to add the diagonal line on a ROC curve plot from PRROC? [migrated]
Here is a sample of ROC curve in PRROC
...
0
votes
1answer
35 views
How to decide threshold values for AUC?
With respect to the pROC package for R(https://rdrr.io/cran/pROC/man/ggroc.html).
...
2
votes
0answers
18 views
Prove that AUROC is an improper scoring rule [duplicate]
It has been stated in many places that AUROC is an improper scoring rule.But I haven't seen anyone proving it. Does someone have a working example that shows that maximizing AUROC actually moves away ...
0
votes
1answer
39 views
1
vote
0answers
24 views
When do ROC-Curves intersect?
In books I see ROC curves like this:
So I always think the model with the highest AUC is better at any threshold. Is this true in general?
When applying usual classification models, can there be ...
2
votes
1answer
32 views
glmStepAIC model is doing better that other models
I am training a model on an imbalanced dataset (about 5-20% of positive class) and trying out different algorithms in R using caret package.
I have 57 predictors and around 2000-3000 observations in ...
0
votes
2answers
65 views
Calculate AUC using sensitivity and specificity values only
How to calculate AUC, if I have values of sensitivity and specificity for various threshold cutoffs?
I have sensitivity and specificity values for 100 thresholds.
...
0
votes
0answers
24 views
3D ROC surface from multinomial logistic regression with more than one predictor
How do you construct a 3D ROC surface in R from a multinomial logistic regression? I have been able to form a 3D ROC surface for the individual models within a multinomial logistic regression, but not ...
0
votes
2answers
38 views
Compare two Roc curves with same Auc
I have 2 churn prediction model.
Both provided very similar AUC values for Roc but with different shape.
How should I assess which to choose based on that fact?
2
votes
2answers
88 views
PR AUC < 50% with ROC AUC > 90% - model good or bad?
I understand for highly imbalanced dataset - we need to look for precision-recall vs ROC AUC to better judge the model.
My question is what is the range for PR AUC below which the model is bad? My ...
1
vote
0answers
18 views
AUC-like measure for multiple simultaneous classification tasks?
I know that given an ordered set of binary labels, and equally-sized ordered set of scalar predictions, we can quantify how cleanly the predictions separate the labels into clean buckets of 0's and 1'...
1
vote
1answer
33 views
ROC-style curves for calculating sample size, power, alpha, and effect size
I found an awesome R package called pwr that does all sorts of calculations about sample sizes, power, effect sizes, and so on, and I've been playing.
I have a ...
0
votes
0answers
26 views
ROC curve for a perfect classifier
I have trained a DNN classifier on a balanced dataset which outputs softmax values $p_1$ and $p_0$ corresponding to probabilities of positive (label 1) and negative (label 0) classes, respectively. I ...
2
votes
1answer
112 views
Generate synthetic data given AUC
I'm experimenting with ROC-AUC for binary classification problems. I want to generate synthetic data for a given AUC score. The ...
0
votes
2answers
59 views
Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?
My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking:
Please provide ...
0
votes
0answers
14 views
Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted? [duplicate]
My data has 119 case and We did ROC for x (continuous variable ) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking "Please provide statistical ...
0
votes
1answer
23 views
Area under the ROC curve accuracy [duplicate]
Why is the area under the ROC curve better than raw accuracy as an out-of- sample evaluation metric?
2
votes
1answer
34 views
SVM: achieving ROC curve by varying misclassification costs
Assume that we have SVM model for binary classification with objective function as follows:
$$
min(\frac{1}{2}\omega\cdot\omega +C^{+}\sum_{i|y_{i}=+1}^{n}\xi_{i}\quad+C^{-}\sum_{i|y_{i}=-1}^{n}\xi_{i}...
1
vote
3answers
55 views
How to graph the difference between similar ROC curves
I have a model and the ROC curve it produces, I modified the model and it produced a second ROC curve that is very similar in shape to the first.
If I graph these ROC curves on the same plot, then it ...
0
votes
0answers
17 views
Which is best method to find optimal cut-off for ROC for continuous data?
Besides youden index can anyone tell me which is the best method to find cut- off point for a ROC CURVE for continuous measurements.
0
votes
0answers
24 views
Systematic way to examine good ROC model from different scoring systems
I am attempting to come up with a scoring system to determine whether one is sick or healthy (based on a certain disease) using biomarkers from blood tests. I normalized all of the biomarkers (divided ...
0
votes
1answer
46 views
Combine multiple metrics to evaluate / choose a ML Model
I'm working on a credit binary classification task.For this business is something usual to meassure model's performance from two metrics:
ROC AUC and KS .This sounds reasonable until I have to choose ...
1
vote
1answer
22 views
How can I interprete ROC-curves? (example inside)
I will be writing an exam in machine learning and am preparing for it.
During my studies I encountered ROC-curves but I can't wrap my head around what they actually tell you.
I have the following ROC-...
0
votes
1answer
76 views
Can the ROC AUC of a total test set be larger than the AUC for any subset of some test set partition?
In testing an ML classifier I built, I came across some confusing behavior.
My model is trained on several distinct datasets which I've combined in order to create one total dataset. I constructed ...
0
votes
0answers
13 views
Determining final TPR and FPR in staged classification model
I'm given pictorial dataset with 3 classes where I'm using the following scheme for classification (this scheme was chosen because classes 2 and 3 are close to each other, so this approach yields ...
0
votes
1answer
32 views
calculate p-value from a confidence inverval
I recently stumbled over the note below and I wonder if there was a method to calculate a p-value for two AUC values with 95%-confidence intervals.
Altman Douglas G, Bland J Martin. How to obtain ...
2
votes
2answers
78 views
ROC/AUC: compare the discrimination ability of a single predictor and a model
I would like to compare a risk prediction test (model) with a single predictor (continuous variable):
Let's say I have a risk prediction testA (e.g. a logistic regression model) that gives a risk ...
0
votes
1answer
39 views
Quasi/Complete separation due to huge and infinite values
(R statistics)
My question is regarding this warning. My data contains patients and healthy subjects. Exponential decay is my outcome measure.
I have a example dataset here
I managed to run ...
0
votes
1answer
42 views
What does this ROC value mean?
What does this roc value mean? How do I interpret it? Are there values which help in inferring it like in case of kappa?
0
votes
0answers
19 views
How can I use receiver operator curve in this SVM classification problem?
Short description of the learning task: I have a corpus containing voice segments annotated with the mean BPM obtained from heart rate recordings. For example, one sample would be like 5 s audio, ...
2
votes
1answer
38 views
Precision-Recall curve interpretation
When given an example confusion matrix:
TP = 5000 FP = 1000
FN = 0 ...
0
votes
0answers
35 views
ROC Curve Evaluation or is that simply Area Under the ROC curve?
I was reading a paper the other day on Machine Learning framework for Sports Performance by Rory P. Bunker and Fadi Thabtah, here for full paper. In it, the authors said:
There is unlikely to be a ...
0
votes
0answers
24 views
Bias of ROC curve
I am trying to use ROC curve with nonparmetric technique ($ROC_m)$. but 'm using following estimates of $\hat f$ and $\hat g$.$$\hat f=\frac {e^ -\frac{(x-t)}{√(h_x)}}{√(h_x ) \left(1+e^ -\frac{(x-t)}{...
0
votes
0answers
47 views
Specificity decreasing when new features are added to glmnet model for case/control prediction
I'm using glmnet for prediction of case/control, which I created with the function train with additional parameters for cross ...
0
votes
0answers
21 views
Implications of deploying a predictive model overfitting training data but consistent in validation folds (classification)
If a model is build on very dirty data, it is common to not be able to prevent an overfitted result even with rigorous regularization attempts. However, it is also common that some lift-producing ...
1
vote
0answers
51 views
conversion prediction / ROC curve using markov chains for channel attribution
I am currently working on a project on multi-channel attribution, using the channel attribution package from Altomare & Loris (2018), which uses markov chains for attribution. A walk-through of ...
0
votes
0answers
15 views
AUC in multiple reader ROC curves?
can anyone tell me how to go about when I have data from 3 observers for ROC curve analysis? Should I take the mean of the AUC of three ROC curves or mean of the variables to calculate a single ROC?
...
0
votes
0answers
22 views
ROC curve with symmetrical kernel
I am trying to use kernels with ROC curve, and 'm succeed to plot them but now my query is about theoretical grounds, i.e. its bias, var, etc.
I want to evaluate the theorems (1 & 2) in Pulit (...
0
votes
0answers
18 views
How to calculate the reclassification calibration statistic (RC statistic)?
In risk prediction models there is a need to evaluate the added diagnostic value of a new variable (biomarker). Different measures have been suggested, among them there is the reclassification ...
3
votes
4answers
154 views
Where does a ROC curve of a perfect classifier start?
A ROC point is a point with a pair of x and y values in the ROC space where x is 1 – specificity and y is sensitivity and the ROC curve always starts at (0.0, 0.0) and ends at (1.0, 1.0).
What about ...
2
votes
2answers
96 views
Area between the ROC curve and the Random Guessing Line
How close is my classifier to random guessing?
I need to quantify the inability of a binary classifier to obtain better results than random guessing in a single number evaluation metric.
The random ...
0
votes
0answers
37 views
obtaining time-dependent survival ROC estimates in test data set
I have survival times and fixed baseline values of biomarkers and risk factors ( I actually have them measured longitudinally but want to deal with the baseline values first).
I'd like to (ideally ...
0
votes
1answer
126 views
ROC-AUC score in sklearn
I'm trying to understand ROC-curve and AUC characteristic for it and found that behaviour of sklearn function roc_auc_score ...
1
vote
0answers
109 views
Does a low Area Under Curve (AUC) for ROC imply worthless classifier? [closed]
I am trying to do a binary classification (0 and 1), and in this case, the end goal is to maximise the True Positives (i.e. maximum no. of 1s should be predicted correctly, even if it gives rise to ...
0
votes
1answer
50 views
Given the following table of class probabilities, find the TPR and FPR at thresholds 0.78 and 0.53
Below is a table of tuples that contain class probabilities for a ROC curve. How can I compute the TPR (True Positive Rate) and FPR (False Positive Rate) at the thresholds 0.78 and 0.53?
Note: the ...
