Questions tagged [roc]
Receiver Operating Characteristic, also known as the ROC curve.
1
vote
1answer
21 views
Why is cross_val_score substantially lower than .score or roc_auc_score?
I have a trained model, a GradientBoostingClassifier. My dataset is 60 thousand something rows of data that I've split into 66/33 train/test sets. Scoring the model via the ...
0
votes
0answers
7 views
finding a cutoff for one predictor from a multivariate logistic regression
I have a disease outcome (true or false of a disease), and several predictors which can be confounding. One of the predictors is a continuous variable, and is considered in the literature to be of ...
1
vote
1answer
12 views
Is it possible for a variable that was proved to be significant from the two-sample t-test to have ROC curve that is close to or below the line x=y?
Say we have a large sample size for each of the two groups, so that the central limit theorem can be applied and thus t-test to compare the two groups means can be justified.
Say group mean ...
0
votes
2answers
31 views
Evaluating Classifiers k fold CV or ROC
I've been doing a project to determine the 'best' classifier for classification on a dataset from UCI.
I used 10 fold stratified cross validation to calculate the mean accuracy. However it was ...
2
votes
1answer
17 views
Adjust ROC analysis for multiple testing?
we did an exploratory prospective study in medicine in order to find parameters which are able to predict a specific post-surgical event (0/1) before the actual surgery. We have about 10 parameters ...
0
votes
1answer
31 views
Determine thresholds for test from ROC-curve
I'm trying to determine the threshold from my original variable from an ROC curve. I have generated the curve using the variable and outcome, and I have generated threshold data from sklearns ROC ...
0
votes
1answer
47 views
How to distinguish overfitting and underfitting from the ROC AUC curve?
For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model.
But how to distinguish whether a ...
2
votes
0answers
27 views
ROC curve for comparing probability of default models
I'm trying to compare two different probability of default models together by roc curve.I calculated the PD for 8 company by two different models.I know about the basic of roc curve and i can ...
2
votes
1answer
23 views
Which metric to use in an ordering problem? auPR / ROC / Lift?
I need to order Users from most likely to perform a binary action X in the next n days, to ...
2
votes
0answers
28 views
Validation ROC AUC not improving with validation cross-entropy loss?
I am training a neural network that is doing binary image classification on several thousand images. I am running 5 fold cross validation (train on 4, validate on 1) with cross entropy (CE) loss. I am ...
0
votes
1answer
37 views
ROC and PR curves after over/under sampling in Unbalanced datasets
As I understood till now, ROC curves are not a good presentation of unbalanced datasets and PR curves are preferred because ROC curves are not sensitive to false positives. If we now use resample ...
1
vote
1answer
21 views
AUC ROC when one class consists of smaller subclasses
This question is different from Binary classification when one class consists of multiple subclasses
I have two classes that I want to distinguish by a supervised learning classifier such as a random ...
0
votes
0answers
19 views
Evaluating binary classifier model. What can say precision, recall etc.? [duplicate]
i'm trying to understand wether my model has good performance or not.
I have binary classifier for summarization sentences: important or not (extractive approach) on specific corpus.
Dataset is ...
2
votes
1answer
73 views
Does AUC/ROC curve return a p-value?
When reading this article, I noticed that the legend in Figure 3 gives a p-value for each AUC (Area Under the Curve) from the ROC (Receiver Operator Characteristic) curves. It says:
The area under ...
0
votes
0answers
47 views
How can I cross-validate a simple binary classifier?
I have a dataset of 30 observations of two variables (one is a class and it's binary, the other is a percentage and it's continuous). My ultimate goal is to build a classifier that is able to predict ...
1
vote
1answer
32 views
Plotting ROC curve in R
This may be a trivial question but I cant answer it myself.
Suppose we have clinical data for patients and healthy controls. how can we draw an ROC curve in R?
...
0
votes
0answers
61 views
Goodness of fit by Hosmer-Lemeshow test and ROC Curve for Logistic Regression not accompanying results conclusions
I am trying to perform Logistic regression on the sample data set. After its modeling, I tried to check its goodness of fit using the Hosmer Lemeshow test and found the p-value < 0.05, which tells ...
1
vote
1answer
86 views
How do I calculate AUC with leave-one-out CV
In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0
votes
0answers
23 views
Decrease in False Positive Rate when True Positive Rate increases (ROC curve analysis)
I am trying to plot a ROC curve for classifier performance varied with size of input data set (effect of data augmentation on testing accuracy).
With an increase of data size, the classifier is ...
2
votes
0answers
38 views
My roc is low while precision and recall are high.Why is it so?
I bulit a naive bayes classifier from 60k vectors of text and each of the text is a 2000 dimension vector(Used bag of words for vectorization). Used simple cross validator to find the best alpha and ...
0
votes
2answers
34 views
threshold cutoff value from ROC for test set evaluation, do I use the cutoff from test ROC or training ROC
Let's say I am doing logistic regression.
I split my data into training and test.
I get an ROC for my training data and it has a cut-off of 0.25
I calculate my evaluation metrics, let's say just ...
0
votes
0answers
41 views
hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R
I would like to calculate the variance of the AUC of readers (for each reader and averaged results) giving a score(1-5) to ...
0
votes
0answers
11 views
Correlation of fitness vs performance on an ensemble model analysis
I have data in the form (the following are just dumb values I use for presentation purposes):
...
2
votes
1answer
73 views
Reason for higher AUC from a test set than a training set using a random forest
I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced ...
1
vote
1answer
28 views
Why are ROC curves and AUC values not always relevant?
So I have read in posts and in literature (Frank Harrel - Regression Modeling
Strategies fx) that depending on what you do, ROC curves and AUC values are not always relevant, but often written in ...
0
votes
0answers
23 views
Understanding model performance
I built a multiclass (and very imbalanced classes) classifier. When evaluating I found an average F1 of .98 and the classifier seems to be working rather well. However, on evaluating the ROC and ...
1
vote
0answers
23 views
What is the FROC (free-response receiver operating characteristic) curve?
I've found it in this paper but can't find any publicly available definition of it. There, it is used to evaluate the performance of aggregate CNNs.
0
votes
0answers
17 views
clustered multi-reader multi-case/modality ROC analysis in R
My problem
I am looking for an R approach to:
do a ROC-Analysis of (clustered (i.e. several observations per case, e.g. prostate segments) / multi reader / multi case/motality) rating (score 1-5) ...
1
vote
1answer
26 views
ROC curve: models have different FPR ranges
I've created ROC curves by calculating the TPR and FPR at various thresholds. The FPR range differs between models, so I'm wondering if AUC is still a valid way to compare the curves. A curve will ...
1
vote
1answer
63 views
How do iso accuracy line related to ROC curve
I have read many articles about ROC curve. Some specified a method to calculate the accuracy of a classifier using iso accuracy lines in a convex hull.
Just like in this articles :
http://mlwiki.org/...
0
votes
0answers
25 views
Best performance metric for highly imbalanced dataset f1 score vs kappa vs AUROC
I have highly imbalanced data (like fraud detection). I usually use f1 score to evaluate model performance. But I also saw people recommend AUROC and cohen's kappa. I'm seeking expert opinion on what ...
0
votes
0answers
27 views
Finding the type II error given the type I error for a minimax decision rule with 0-1 loss
Assume a two world state ($\Omega=\left\{ \omega_{0},\omega_{1}\right\}$ ) scenario and that we are given the [continuous] ROC curve $\left\{ \left(\alpha\left(\theta\right),1-\beta\left(\theta\right)\...
2
votes
2answers
116 views
Why AUC is not a good performance metric for a classification model?
After understanding the benefits of AUC
I was stumbled to know that in some scenarios it might not be a good performance metric for evaluating a classification model. The below are the 2 scenarios:
...
0
votes
1answer
47 views
Finding a cut-off point
I'm trying to find a statistically significant cut-off point for number of attempts after which the risk of a poor outcome significantly increases.
I'm using SPSS. First I generated a ROC curve in ...
0
votes
0answers
10 views
Why is one of the ROC curve generated by kfold always to the right?
Across multiple execution of this code:
...
1
vote
1answer
29 views
Gini above 1 when bootstrapping
Let's say I have a dataset (data), which contains the binary target variable class and the predictions (probabilities in [0,1]).
...
1
vote
0answers
59 views
Using An ROC Curve to Evaluate a model
I have a number of questions on the ROC curves when being used to evaluate a model.
My understanding of them is they can be used to determine the probability cutoff when classifying a row in a dataset ...
2
votes
1answer
66 views
Why are the trivial points included when calculating AUC?
I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue ...
0
votes
2answers
798 views
What is the formula to calculate the area under the ROC curve from a contingency table?
For example, if my table is:
...
2
votes
1answer
164 views
How to Interpret AUROC score?
My model has an AUROC value of 0.7, and I have a 75:25 class (75% negative, 25% positive) imbalance. From my understanding, AUROC is calculated by using different thresholds for considering the ...
0
votes
0answers
29 views
Insignificant t test or MW U test yet high ROC AUC and vice versa
In SPSS, when I conduct a Student's t test or Mann-Whitney U test on (lots of) variables when comparing between 2 groups, some differences are denoted significant, and others aren't.
When I conduct ...
2
votes
1answer
152 views
pROC versus ROCR
This is a very basic question but I don't get why the following provides different results when applying pROC or ROCR, see plot.
...
1
vote
1answer
136 views
must ROC curve be concave?
How can I decide whether the attached is a correct ROC curve?
The underlying algorithm consists in calculating an ODE system and storing the number of certain cancer cells calculated by this model ...
0
votes
2answers
36 views
any intuitive explanation why all classifiers converge to (1,1) point on ROC?
Although I think I understand the (1,1) point on ROC, meaning a classifier that unconditionally issues positive (or negative) classification for all tests in a binary classification scenario, I have ...
2
votes
1answer
87 views
Classification model accuracy, roc auc score, f1 score 100%
I am working on a binary classification problem. I have split the train set and when I evaluate the model on the validation set all metrics are 100% which is unrealistic considering that I haven't ...
1
vote
0answers
127 views
Drawing ROC curves from RFE() training results in caret
I want to generate ROC curves using the training data and results from the rfe function in caret. I have managed to do this with the code below but there is some inconsistency between the ROC value ...
0
votes
2answers
41 views
Performance evaluation
I'd like to test the performance of a penalized regression. I did three separate regressions for each response variable (one numerical, one binomial and one multinomial). I was checking this link, and ...
3
votes
0answers
32 views
Confidence intervals of AUC obtained by merging/pooling predictions from different test sets
I have one question regarding the CIs of the AUROC calculated merging/pooling the predictions coming from different test sets.
In one analysis, we use a sort of nested cross-validation approach, ...
1
vote
0answers
31 views
How to make really bad results from a machine learning model better by reversing predictions
I trained a classification model on some data with two classes and have really low accuracy. I have a false-positive rate of 86 % for both classes I am trying to predict. I was wondering if I could ...
1
vote
1answer
101 views
Keras: How do I train a binary classification net where one class is much more frequent then the other?
I want to train a binary classification net (for NLP) where one class is much more frequent then the other (using Keras).
I have learned that in this case (one class is much more frequent then the ...
