Questions tagged [roc]
Receiver Operating Characteristic, also known as the ROC curve.
0
votes
1answer
30 views
What does this ROC value mean?
What does this roc value mean? How do I interpret it? Are there values which help in inferring it like in case of kappa?
0
votes
0answers
13 views
How can I use receiver operator curve in this SVM classification problem?
Short description of the learning task: I have a corpus containing voice segments annotated with the mean BPM obtained from heart rate recordings. For example, one sample would be like 5 s audio, ...
2
votes
1answer
27 views
Precision-Recall curve interpretation
When given an example confusion matrix:
TP = 5000 FP = 1000
FN = 0 ...
0
votes
0answers
32 views
ROC Curve Evaluation or is that simply Area Under the ROC curve?
I was reading a paper the other day on Machine Learning framework for Sports Performance by Rory P. Bunker and Fadi Thabtah, here for full paper. In it, the authors said:
There is unlikely to be a ...
0
votes
0answers
19 views
Bias of ROC curve
I am trying to use ROC curve with nonparmetric technique ($ROC_m)$. but 'm using following estimates of $\hat f$ and $\hat g$.$$\hat f=\frac {e^ -\frac{(x-t)}{√(h_x)}}{√(h_x ) \left(1+e^ -\frac{(x-t)}{...
0
votes
0answers
39 views
Specificity decreasing when new features are added to glmnet model for case/control prediction
I'm using glmnet for prediction of case/control, which I created with the function train with additional parameters for cross ...
0
votes
0answers
18 views
Implications of deploying a predictive model overfitting training data but consistent in validation folds (classification)
If a model is build on very dirty data, it is common to not be able to prevent an overfitted result even with rigorous regularization attempts. However, it is also common that some lift-producing ...
0
votes
0answers
18 views
conversion prediction / ROC curve using markov chains for channel attribution
I am currently working on a project on multi-channel attribution, using the channel attribution package from Altomare & Loris (2018), which uses markov chains for attribution. A walk-through of ...
0
votes
0answers
11 views
AUC in multiple reader ROC curves?
can anyone tell me how to go about when I have data from 3 observers for ROC curve analysis? Should I take the mean of the AUC of three ROC curves or mean of the variables to calculate a single ROC?
...
0
votes
0answers
22 views
ROC curve with symmetrical kernel
I am trying to use kernels with ROC curve, and 'm succeed to plot them but now my query is about theoretical grounds, i.e. its bias, var, etc.
I want to evaluate the theorems (1 & 2) in Pulit (...
0
votes
0answers
16 views
How to calculate the reclassification calibration statistic (RC statistic)?
In risk prediction models there is a need to evaluate the added diagnostic value of a new variable (biomarker). Different measures have been suggested, among them there is the reclassification ...
3
votes
4answers
127 views
Where does a ROC curve of a perfect classifier start?
A ROC point is a point with a pair of x and y values in the ROC space where x is 1 – specificity and y is sensitivity and the ROC curve always starts at (0.0, 0.0) and ends at (1.0, 1.0).
What about ...
2
votes
2answers
73 views
Area between the ROC curve and the Random Guessing Line
How close is my classifier to random guessing?
I need to quantify the inability of a binary classifier to obtain better results than random guessing in a single number evaluation metric.
The random ...
0
votes
0answers
14 views
obtaining time-dependent survival ROC estimates in test data set
I have survival times and fixed baseline values of biomarkers and risk factors ( I actually have them measured longitudinally but want to deal with the baseline values first).
I'd like to (ideally ...
0
votes
1answer
30 views
ROC-AUC score in sklearn
I'm trying to understand ROC-curve and AUC characteristic for it and found that behaviour of sklearn function roc_auc_score ...
1
vote
0answers
72 views
Does a low Area Under Curve (AUC) for ROC imply worthless classifier? [closed]
I am trying to do a binary classification (0 and 1), and in this case, the end goal is to maximise the True Positives (i.e. maximum no. of 1s should be predicted correctly, even if it gives rise to ...
0
votes
1answer
36 views
Given the following table of class probabilities, find the TPR and FPR at thresholds 0.78 and 0.53
Below is a table of tuples that contain class probabilities for a ROC curve. How can I compute the TPR (True Positive Rate) and FPR (False Positive Rate) at the thresholds 0.78 and 0.53?
Note: the ...
1
vote
0answers
13 views
How to combine from multiple probability in adaboost? [closed]
I tried to implement adaboost, then I want to create ROC and count for the AUROC. I use tree as my base classifier. I got the probability from each tree. How to combine them?
For simplicity, there ...
12
votes
5answers
2k views
Philosophical question on logistic regression: why isn't the optimal threshold value trained?
Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold ...
0
votes
0answers
26 views
How to interpret ROC curve? [duplicate]
I am currently doing a classification problem for classifying the functional class and non-functional class of peptidase cleavage site. The data on non-functional class (negative class) is highly ...
0
votes
1answer
46 views
AUC of single model vs AUC of separate models on same data
I have created two separate binary classifiers that predict the same kind of label using 2 separate datasets. The data is in the same format. They both have a AUC of 0.94 and 0.95
I have then created ...
1
vote
0answers
32 views
Estimating the False Reject Rate (FRR) of a classifier in production
I have trained a binary classifier which runs in production on remote distributed devices (which are out of my control).
The model was trained on positive and negative samples, and I have chosen the ...
1
vote
0answers
10 views
Is there a way to intuitively think about AUC associated with an ROC plot in the case where our predictions are binary?
Suppose we have a true set of labels, which are either $0,1$, and we have a set of predicted labels that are also either $0,1$. In this case, a possible ROC plot has one point connecting the line, ...
1
vote
0answers
17 views
What is the rationale for assuming that prediction values of a classifier are normally distributed per class?
A standard image to describe how to understand ROC curves is by showing the distribution of a model's predictions, grouped by real label. In this image, a histogram of predictions for class 'good' (in ...
0
votes
1answer
25 views
Is it valid to use ROC calculated during test/validation to interpret results of final production model?
I've trained a binary classification model which outputs a "probability" between (0,1).
During testing and validation, I use the ROC to measure the performance of the model. Also, I use the ROC to ...
1
vote
2answers
85 views
AUPRC vs AUROC and updating training set in quasi-classification problem
I have an unbalanced classification problem (95% "0", 5% "1") regarding quality control."0" means "no problem" and "1" means "problem".
I'm not predicting real cases one by one, this is, my client ...
0
votes
1answer
44 views
Stacking AUC vs. average over folds
I have a two class prediction problem where in one class I have 70% of the samples and in the other class 30% of the samples, so class imbalance. I'm conducting 10-fold cross-validation. To calcualte ...
0
votes
0answers
50 views
evaluating logistic regression's performance
I am working on the scoring model and I aim to predict the probability of default. I have, say m, different candidate Logistic Regression models $M_{1}, \dots, M_{m}$ and I would like to choose the ...
0
votes
0answers
41 views
Is it possible to calculate the ROC_AUC score for a class in a multiclass problem that is not in the predicted array?
I was working on calculating the auc score for a multiclass problem and came across this problem.
Suppose I have a data set with three classes [0,1,2]
My test set is like this [0,1,2,0,0,2]
My ...
0
votes
0answers
31 views
How Gini/AUC of two features is bounded by individual features?
Consider binary classification problem and popular quality measure ROC AUC (which is almost the same as Gini coefficient G = 2*AUC - 1 ).
Assume we have two features F1, F2.
Question (rough version)...
0
votes
0answers
30 views
Cross validation scores better than training score when using ROC_AUC (but are the same when accuracy, F1 etc scores are used)
I'm running a logistic regression on a balanced data set and wanted to validate my model using the ROC-AUC metric.
...
0
votes
1answer
34 views
Using predicted probabilities from logistic regression as dependent variable in a linear regression
I'm trying to run a Response Surface Analysis in SAS, but this is only possible with a continuous outcome, whereas my outcome variable is binary. I got the advice to first run a logistic regression ...
3
votes
2answers
130 views
Why is a PR curve considered better than an ROC curve for imbalanced datasets?
I have heard from multiple sources that a precision-recall curve is considered better than an ROC curve when testing a classifier on a dataset with a class imbalance.
https://www.biostat.wisc.edu/~...
0
votes
1answer
41 views
Comparing two binary classifiers
I have 2 binary classifiers and a test set.
For the first of the classifiers I can compute any metrics for any value of a threshold, e.g. I can plot ROC curve and calculate precision, recall, F1 etc ...
0
votes
1answer
61 views
Why is there a loop in the ROC curve? [closed]
I am trying to implement a multi-criteria classifier algorithm. It uses 6 criteria that output 1 or 0 if a specific signal is detected in the input data or not.
Then it computes a weighted average of ...
0
votes
0answers
50 views
Do you need to adjust the probability if you use the 'class_weight' parameter in LogisticRegression-sklearn?
I have a imbalanced dataset and I want the the output as probabilities and not labels. Hence using Logistic Regression seemed to be the obvious choice.
However the classsifer started predicting all ...
1
vote
0answers
45 views
Aggregating ROC AUC values of several Logistic Regression Models
I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
2
votes
1answer
44 views
Area Under the Curve - Variable and Log Transformed Variable
I have a situation where I am fitting two simple logistic regression models - one model with the variable of interest included as the only predictor, and the other model with the log of the variable ...
0
votes
1answer
39 views
Is it correctly understood that ROC/AUC cannot be calculated without flexible criterion value?
I have a proprietary predictor that simply gives me a binary output. Let's say that it is detecting faulty units. In a set where 27 units are faulty and 76 units are working the predictor correctly ...
0
votes
0answers
13 views
Why is the AUC when plotted against the wilcoxon rank sum p-value, result in a plot that is not linear?
I have a binary response variable with many continuous predictor variables (about 5000). I first computed the AUC for each predictor variable, then computed the p-value associated with the AUC using ...
0
votes
0answers
10 views
using survival censoring indicator as a binary outcome for ROC curves and logistic regression
I wonder how acceptable it is, pros and cons, of using the censoring indicator with survival data as a binary outcome for ROC curves and logistic regression.
One issue is if we have early dropout / ...
2
votes
2answers
64 views
Why does pROC roc work with non-probability predictions?
With the pROC package, I can do this:
true <- c(1, 1, 1, 0)
predicted <- c(0.5, 0.1, 0.6, 0.1)
roc(true, predicted)
which gives as expected:
...
0
votes
1answer
52 views
High AUC and Accuracy but weird output in confusion matrix
I am working on image classification problem to determine gender given a face. The dataset is located here gender face dataset on kaggle (link to my notebook). The class distribution is as follows.
<...
0
votes
0answers
44 views
ROC curve for multi-variable based prediction in a 3 class classification
I have a data with 10 variables (continuous with log transformed values) that I am using to accurately predict in a 3 class classification.
I used RF model to select those 10 variables by first ...
1
vote
1answer
144 views
Why is cross_val_score substantially lower than .score or roc_auc_score?
I have a trained model, a GradientBoostingClassifier. My dataset is 60 thousand something rows of data that I've split into 66/33 train/test sets. Scoring the model via the ...
0
votes
0answers
13 views
finding a cutoff for one predictor from a multivariate logistic regression
I have a disease outcome (true or false of a disease), and several predictors which can be confounding. One of the predictors is a continuous variable, and is considered in the literature to be of ...
1
vote
1answer
14 views
Is it possible for a variable that was proved to be significant from the two-sample t-test to have ROC curve that is close to or below the line x=y?
Say we have a large sample size for each of the two groups, so that the central limit theorem can be applied and thus t-test to compare the two groups means can be justified.
Say group mean ...
0
votes
2answers
42 views
Evaluating Classifiers k fold CV or ROC
I've been doing a project to determine the 'best' classifier for classification on a dataset from UCI.
I used 10 fold stratified cross validation to calculate the mean accuracy. However it was ...
2
votes
1answer
42 views
Adjust ROC analysis for multiple testing?
we did an exploratory prospective study in medicine in order to find parameters which are able to predict a specific post-surgical event (0/1) before the actual surgery. We have about 10 parameters ...
0
votes
1answer
538 views
Determine thresholds for test from ROC-curve
I'm trying to determine the threshold from my original variable from an ROC curve. I have generated the curve using the variable and outcome, and I have generated threshold data from sklearns ROC ...
