Questions tagged [roc]
Receiver Operating Characteristic, also known as the ROC curve.
4
votes
3answers
162 views
Philosophical question on logistic regression: why isn't the optimal threshold value trained?
Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold ...
0
votes
0answers
16 views
How to create a ROC curve for a model which has log of odds as response? [on hold]
I have a question on plotting ROC curve for my model which has log of odds as the response. For example:
model<-lm((ln(y/1-y)~Temp+RH+DmaxT, data=fit) #'y' is a proportion
Predicted response was ...
0
votes
0answers
11 views
How to interpret ROC curve? [duplicate]
I am currently doing a classification problem for classifying the functional class and non-functional class of peptidase cleavage site. The data on non-functional class (negative class) is highly ...
0
votes
1answer
34 views
AUC of single model vs AUC of separate models on same data
I have created two separate binary classifiers that predict the same kind of label using 2 separate datasets. The data is in the same format. They both have a AUC of 0.94 and 0.95
I have then created ...
1
vote
0answers
21 views
Estimating the False Reject Rate (FRR) of a classifier in production
I have trained a binary classifier which runs in production on remote distributed devices (which are out of my control).
The model was trained on positive and negative samples, and I have chosen the ...
1
vote
0answers
8 views
Is there a way to intuitively think about AUC associated with an ROC plot in the case where our predictions are binary?
Suppose we have a true set of labels, which are either $0,1$, and we have a set of predicted labels that are also either $0,1$. In this case, a possible ROC plot has one point connecting the line, ...
1
vote
0answers
17 views
What is the rationale for assuming that prediction values of a classifier are normally distributed per class?
A standard image to describe how to understand ROC curves is by showing the distribution of a model's predictions, grouped by real label. In this image, a histogram of predictions for class 'good' (in ...
0
votes
1answer
21 views
Is it valid to use ROC calculated during test/validation to interpret results of final production model?
I've trained a binary classification model which outputs a "probability" between (0,1).
During testing and validation, I use the ROC to measure the performance of the model. Also, I use the ROC to ...
1
vote
2answers
54 views
AUPRC vs AUROC and updating training set in quasi-classification problem
I have an unbalanced classification problem (95% "0", 5% "1") regarding quality control."0" means "no problem" and "1" means "problem".
I'm not predicting real cases one by one, this is, my client ...
0
votes
1answer
26 views
Stacking AUC vs. average over folds
I have a two class prediction problem where in one class I have 70% of the samples and in the other class 30% of the samples, so class imbalance. I'm conducting 10-fold cross-validation. To calcualte ...
0
votes
0answers
36 views
evaluating logistic regression's performance
I am working on the scoring model and I aim to predict the probability of default. I have, say m, different candidate Logistic Regression models $M_{1}, \dots, M_{m}$ and I would like to choose the ...
0
votes
0answers
10 views
Is it possible to calculate the ROC_AUC score for a class in a multiclass problem that is not in the predicted array?
I was working on calculating the auc score for a multiclass problem and came across this problem.
Suppose I have a data set with three classes [0,1,2]
My test set is like this [0,1,2,0,0,2]
My ...
0
votes
0answers
22 views
How Gini/AUC of two features is bounded by individual features?
Consider binary classification problem and popular quality measure ROC AUC (which is almost the same as Gini coefficient G = 2*AUC - 1 ).
Assume we have two features F1, F2.
Question (rough version)...
0
votes
0answers
18 views
Cross validation scores better than training score when using ROC_AUC (but are the same when accuracy, F1 etc scores are used)
I'm running a logistic regression on a balanced data set and wanted to validate my model using the ROC-AUC metric.
...
0
votes
1answer
30 views
Using predicted probabilities from logistic regression as dependent variable in a linear regression
I'm trying to run a Response Surface Analysis in SAS, but this is only possible with a continuous outcome, whereas my outcome variable is binary. I got the advice to first run a logistic regression ...
3
votes
2answers
75 views
Why is a PR curve considered better than an ROC curve for imbalanced datasets?
I have heard from multiple sources that a precision-recall curve is considered better than an ROC curve when testing a classifier on a dataset with a class imbalance.
https://www.biostat.wisc.edu/~...
0
votes
1answer
35 views
Comparing two binary classifiers
I have 2 binary classifiers and a test set.
For the first of the classifiers I can compute any metrics for any value of a threshold, e.g. I can plot ROC curve and calculate precision, recall, F1 etc ...
0
votes
1answer
54 views
Why is there a loop in the ROC curve? [closed]
I am trying to implement a multi-criteria classifier algorithm. It uses 6 criteria that output 1 or 0 if a specific signal is detected in the input data or not.
Then it computes a weighted average of ...
0
votes
0answers
22 views
Do you need to adjust the probability if you use the 'class_weight' parameter in LogisticRegression-sklearn?
I have a imbalanced dataset and I want the the output as probabilities and not labels. Hence using Logistic Regression seemed to be the obvious choice.
However the classsifer started predicting all ...
0
votes
0answers
29 views
Aggregating ROC AUC values of several Logistic Regression Models
I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
2
votes
1answer
35 views
Area Under the Curve - Variable and Log Transformed Variable
I have a situation where I am fitting two simple logistic regression models - one model with the variable of interest included as the only predictor, and the other model with the log of the variable ...
0
votes
1answer
32 views
Is it correctly understood that ROC/AUC cannot be calculated without flexible criterion value?
I have a proprietary predictor that simply gives me a binary output. Let's say that it is detecting faulty units. In a set where 27 units are faulty and 76 units are working the predictor correctly ...
0
votes
0answers
12 views
Why is the AUC when plotted against the wilcoxon rank sum p-value, result in a plot that is not linear?
I have a binary response variable with many continuous predictor variables (about 5000). I first computed the AUC for each predictor variable, then computed the p-value associated with the AUC using ...
0
votes
0answers
7 views
using survival censoring indicator as a binary outcome for ROC curves and logistic regression
I wonder how acceptable it is, pros and cons, of using the censoring indicator with survival data as a binary outcome for ROC curves and logistic regression.
One issue is if we have early dropout / ...
2
votes
2answers
33 views
Why does pROC roc work with non-probability predictions?
With the pROC package, I can do this:
true <- c(1, 1, 1, 0)
predicted <- c(0.5, 0.1, 0.6, 0.1)
roc(true, predicted)
which gives as expected:
...
0
votes
1answer
40 views
High AUC and Accuracy but weird output in confusion matrix
I am working on image classification problem to determine gender given a face. The dataset is located here gender face dataset on kaggle (link to my notebook). The class distribution is as follows.
<...
0
votes
0answers
41 views
ROC curve for multi-variable based prediction in a 3 class classification
I have a data with 10 variables (continuous with log transformed values) that I am using to accurately predict in a 3 class classification.
I used RF model to select those 10 variables by first ...
1
vote
1answer
87 views
Why is cross_val_score substantially lower than .score or roc_auc_score?
I have a trained model, a GradientBoostingClassifier. My dataset is 60 thousand something rows of data that I've split into 66/33 train/test sets. Scoring the model via the ...
0
votes
0answers
10 views
finding a cutoff for one predictor from a multivariate logistic regression
I have a disease outcome (true or false of a disease), and several predictors which can be confounding. One of the predictors is a continuous variable, and is considered in the literature to be of ...
1
vote
1answer
13 views
Is it possible for a variable that was proved to be significant from the two-sample t-test to have ROC curve that is close to or below the line x=y?
Say we have a large sample size for each of the two groups, so that the central limit theorem can be applied and thus t-test to compare the two groups means can be justified.
Say group mean ...
0
votes
2answers
37 views
Evaluating Classifiers k fold CV or ROC
I've been doing a project to determine the 'best' classifier for classification on a dataset from UCI.
I used 10 fold stratified cross validation to calculate the mean accuracy. However it was ...
2
votes
1answer
28 views
Adjust ROC analysis for multiple testing?
we did an exploratory prospective study in medicine in order to find parameters which are able to predict a specific post-surgical event (0/1) before the actual surgery. We have about 10 parameters ...
0
votes
1answer
211 views
Determine thresholds for test from ROC-curve
I'm trying to determine the threshold from my original variable from an ROC curve. I have generated the curve using the variable and outcome, and I have generated threshold data from sklearns ROC ...
0
votes
1answer
179 views
How to distinguish overfitting and underfitting from the ROC AUC curve?
For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model.
But how to distinguish whether a ...
2
votes
0answers
31 views
ROC curve for comparing probability of default models
I'm trying to compare two different probability of default models together by roc curve.I calculated the PD for 8 company by two different models.I know about the basic of roc curve and i can ...
2
votes
1answer
30 views
Which metric to use in an ordering problem? auPR / ROC / Lift?
I need to order Users from most likely to perform a binary action X in the next n days, to ...
3
votes
0answers
120 views
Validation ROC AUC not improving with validation cross-entropy loss?
I am training a neural network that is doing binary image classification on several thousand images. I am running 5 fold cross validation (train on 4, validate on 1) with cross entropy (CE) loss. I am ...
0
votes
1answer
49 views
ROC and PR curves after over/under sampling in Unbalanced datasets
As I understood till now, ROC curves are not a good presentation of unbalanced datasets and PR curves are preferred because ROC curves are not sensitive to false positives. If we now use resample ...
1
vote
1answer
36 views
AUC ROC when one class consists of smaller subclasses
This question is different from Binary classification when one class consists of multiple subclasses
I have two classes that I want to distinguish by a supervised learning classifier such as a random ...
0
votes
0answers
19 views
Evaluating binary classifier model. What can say precision, recall etc.? [duplicate]
i'm trying to understand wether my model has good performance or not.
I have binary classifier for summarization sentences: important or not (extractive approach) on specific corpus.
Dataset is ...
2
votes
1answer
231 views
Does AUC/ROC curve return a p-value?
When reading this article, I noticed that the legend in Figure 3 gives a p-value for each AUC (Area Under the Curve) from the ROC (Receiver Operator Characteristic) curves. It says:
The area under ...
0
votes
0answers
60 views
How can I cross-validate a simple binary classifier?
I have a dataset of 30 observations of two variables (one is a class and it's binary, the other is a percentage and it's continuous). My ultimate goal is to build a classifier that is able to predict ...
1
vote
1answer
56 views
Plotting ROC curve in R
This may be a trivial question but I cant answer it myself.
Suppose we have clinical data for patients and healthy controls. how can we draw an ROC curve in R?
...
0
votes
0answers
250 views
Goodness of fit by Hosmer-Lemeshow test and ROC Curve for Logistic Regression not accompanying results conclusions
I am trying to perform Logistic regression on the sample data set. After its modeling, I tried to check its goodness of fit using the Hosmer Lemeshow test and found the p-value < 0.05, which tells ...
1
vote
1answer
164 views
How do I calculate AUC with leave-one-out CV
In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0
votes
0answers
41 views
Decrease in False Positive Rate when True Positive Rate increases (ROC curve analysis)
I am trying to plot a ROC curve for classifier performance varied with size of input data set (effect of data augmentation on testing accuracy).
With an increase of data size, the classifier is ...
2
votes
0answers
43 views
My roc is low while precision and recall are high.Why is it so?
I bulit a naive bayes classifier from 60k vectors of text and each of the text is a 2000 dimension vector(Used bag of words for vectorization). Used simple cross validator to find the best alpha and ...
1
vote
2answers
50 views
threshold cutoff value from ROC for test set evaluation, do I use the cutoff from test ROC or training ROC
Let's say I am doing logistic regression.
I split my data into training and test.
I get an ROC for my training data and it has a cut-off of 0.25
I calculate my evaluation metrics, let's say just ...
0
votes
0answers
44 views
hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R
I would like to calculate the variance of the AUC of readers (for each reader and averaged results) giving a score(1-5) to ...
2
votes
1answer
175 views
Reason for higher AUC from a test set than a training set using a random forest
I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced ...
