Receiver Operating Characteristic, also known as ROC curve.

learn more… | top users | synonyms

1
vote
2answers
36 views

what is the difference between area under roc and weighted area under roc?

Thanks in advance for the help. I have an unbalanced dataset that I am using for a binary classification problem. The classes are unbalanced. I believe that in such a case that weighted area under ...
1
vote
1answer
50 views

Area under ROC given the two distributions

I have two distributions p1 and p2 (generating in R using distr package), and I want to compute the area under the ROC curve. To construct the ROC curve, I have to compute the probability of detection ...
1
vote
2answers
48 views

Relationship between ROC curve and statistical significance in comparison of two groups

My question is about any possible relationship between the significance of a comparison test (Mann-Whitney U) and the ROC curve. If the comparison test is strongly significant, should we expect a ...
0
votes
1answer
40 views

Did I understand ROC curves correctly?

I want to classify objects by their area into two classes. I implemented several area estimates that I want to compare. For each object, I have a gold standard indicating to which of the two classes ...
3
votes
1answer
61 views

ROC curve for two-sided cut-off

I am very, very confused about ROC curves. I have a Bayesian model which outputs a prevalence on a continuous scale between 0 and 1. I have a classification I would like to use that classifies that ...
1
vote
0answers
23 views

Formula to derive probabilities when using 'gbm' method through 'caret' package in R

I am training a classification problem by 'gbm' algorithm through 'caret' package in r.The response variable is a yes/nope type. Here 'objmodel' is the model I trained through method='gbm' and package ...
0
votes
0answers
12 views

What type of curve to use in object detection task to measure performance of detector?

I have an object detection task with one object type(one object type + background = object detection with sliding window as binary classification of each window). And my data is unbalanced(many ...
0
votes
1answer
49 views

The relation between a confusion matrix and a ROC curve

As an example I have a confusion matrix that shows good accuracy but poor performance on sensitivity because of imbalanced classes. I made this fictive table for a presentation. ...
0
votes
1answer
32 views

Calculating AUC for a GEE

I have used the geeglm package to build a GEE that predicts animal activity (a binary response, active or not) from weather data (e.g., Temperature, a continuous variable). TEMPC <- ...
0
votes
0answers
23 views

What is the difference between GINI and AUC curve interpretation?

we used to create GINI curve using lift created with help of percentage of good and bad for scorecard modelling. But what I have studied that ROC curve is created using Confusion matrix with ...
1
vote
1answer
21 views

compute ROC from Sensitivity and Specificity

It might sound like a stupid question, but I've got to ask it. Let's say I have a classifier, with one discrete bounded parameter. I have run the classifier over all possible parameter values, and ...
0
votes
0answers
10 views

ROC Analyses - can collapse assessment time periods?

I hope you can provide some advice on doing ROC analyses. I have been running ROC analyses on several different screening tests and have a question about one of my data sets. It is a longitudinal data ...
2
votes
1answer
24 views

ROC and constant factor on probabilities

I play around with a few data to learn and I am wondering about something; I can evaluate my results with ROC which is processed from FP and FN. I had predicted a few probabilities for my events to ...
0
votes
0answers
57 views

ROC curve cut off and weights

I have a dependent variable distinguishing between patients that should go to treatment A or treatment B. I want to develop a questionnaire containing binary variables that should decide if the ...
0
votes
0answers
10 views

Is there any lower limit for number of positives when generating lift plot?

I am wondering if there is any condition on number of positives in test set when I am trying to compute lift plot to check the properties of my classifier?
9
votes
2answers
469 views

Difference between regression analysis and curve fitting

Can anybody please explain to me the real difference(s) between regression analysis and curve fitting (linear and nonlinear), with an example if possible? It seems that both try to find a ...
0
votes
0answers
31 views

Interpretation and meaningfulness of regression coefficients

I have performed logistic regression on banking data which is trying to predict the bad customers correctly due to the cost involved. I have build a model and pasting a picture of the output obtained ...
0
votes
0answers
13 views

ROC and post estimation COX Harrell's C using your dataset

I have built a predictive model using a combination of logistic and cox regression models. I did it using a dataset of about 5000 records. I would like to calculate the AUC and the Harrell's post ...
2
votes
1answer
44 views

So many significant explanatory variables and so small auc

Have you ever seen a model with almost every significant variable and such small auc (area under the ROC curve) ? What might be the cause of it? When I saw summary of a model I thought this model will ...
0
votes
0answers
37 views

How is the mean area under the curve calculated?

I am using 10-fold cross-validation for performance estimation. From each of the ten iterations, I get an area under the curve (AUC) metric, e.g. ...
0
votes
0answers
50 views

Area under ROC curve vs. Accuracy in unbalanced sample

I have a binary classification problem with 3000 samples (number of 1 as outputs = 300, number of 0 as outputs = 1700). After balancing database (selecting 300 samples from 0 outputs) I trained the ...
1
vote
1answer
42 views

Constructing an ROC curve without true negatives?

I’m not really a statistician by training so please excuse any incorrect terminology or if this is a painfully obvious question. I am trying to assess the performance of two algorithms. I don’t wish ...
3
votes
1answer
74 views

Simple Question about ROC Curve

Motivated by this reference, it states under ROC Space When evaluating a binary classifier, we often use a Confusion Matrix...however here we need only TPR and FPR I'd feel more comfortable if ...
4
votes
1answer
179 views

Different shapes of an ROC curve

What are the possible shapes of an ROC curve? Is it necessary for an ROC curve to be shaped like a normal distribution curve? Can we regard the following two curves as ROC with the area under the ...
0
votes
0answers
24 views

Forward sequential feature selection improving classifier performance?

I was in a bit of a conversation with a co-worker about using forward selection. My training data is on order of ~6,000 w/ dimensionality of 1,200, and testing data on order of ~3,000. Currently, I'm ...
3
votes
1answer
58 views

Getting the optimal age cut off

I am working with a database of diseased and non-diseased patients. I would like to recommend an optimal age cut off for disease screening. The common problem with screening is that it creates a ...
0
votes
0answers
27 views

ROC for optimal cut-off using Svyglm object

I am trying to estimate optimal cut-points via ROC for a complex survey data. I able to achieve this task with ROC(in Epi) and OptimalCutpoints package but for the unweighted sample. The ...
4
votes
1answer
89 views

Model comparison between glm (with Firth correction), random Forest, penalised SVM

I am currently developing three models to classify features of gene sites. I was using glm (with Firth correction), random Forest and SVM to build the models and I used forward and backward ...
1
vote
2answers
69 views

Using ROC curve for balanced data

I understand that using the area under the ROC curve is a useful error measurement for unbalanced data. What happens if we use it for balanced data?
1
vote
0answers
115 views

How can I calculate AUC using Gini coefficient?

In the Gini Coefficient's Wikipedia page, it is defined as $G= 1 - \frac{\Sigma_{i=1}^n f(y_i)(S_{i-1}+S_i)}{S_n}$ for discrete variables, where $S_i= \Sigma_{j=1}^i f(y_i)y_i$ and $S_0=0$ ($y$ being ...
1
vote
0answers
123 views

Calculating two-tailed p-value from z-score for ROC AUC comparison

I am comparing two predictive models by their bootstrapped ROC AUCs with the method originally described by Hanley and McNeil and modified for bootstrapped data by Robin et al. I'm calculating the ...
1
vote
1answer
44 views

ROC curve when the numbers of both total positives and negatives are unknown

I have to use a roc curve to find an agreement between sensitivity between the FPR and TPR. The problem is that I only know the true positives. How could I circumvent this? More accurate, I have a ...
0
votes
1answer
46 views

How to know if my data is balanced or imbalanced for an ROC curve analysis?

I am doing a research on the reliability of different models in detecting hidden defects in a test specimen. I have made a test specimen with defect prevalence about 25% (12 positive out of 49 total ...
0
votes
1answer
49 views

What does ROC-EER in percent stand for?

Ive tried to understand what the ROC Curve represents and what EER (Equal Error Rate) means. And I somehow think I got to understand some of the explanations I read on the internet and videos I ...
6
votes
2answers
838 views

d prime with 100% hit rate probability and 0% false alarm probability

I would like to calculate d prime for a memory task that involves detecting old and new items. The problem I have is that some of the subjects have hit rate of 1 and/or false alarm rate of 0, which ...
3
votes
2answers
174 views

One model performs better than the other. How to measure if it is statistically significant?

So, let's say that I train two models on the same dataset. I run the experiment once and I get the following results: Using a Neural Network I get an AUC ROC of 0.941. Using Random Forest I get an ...
5
votes
1answer
133 views

Is the objective to beat a random classifier when the data set is skewed using PR curves?

I have a testing data set where 1/3 of the observations are class-1 objects and the remainder class-0. Hence, the data set is skewed (skewed classifier), literature suggests that if the data set is ...
1
vote
1answer
95 views

Threshold selection by intersection of Sensitivity and Specificity

Some days ago, I learned in a lecture that the intersection of Sensitivity and Specificity provides an optimal compromise for choosing a classification threshold for logit or probit models. However, ...
0
votes
0answers
26 views

Recommended performance metrics of a binary classifier having an example-based cost matrix

I would like to know what are the recommended performance metrics to assess a binary classifier, when the cost matrix is changing for each sample. My problem comes from the fact that in this case, ...
2
votes
0answers
29 views

does the area under the curve (AUC) has some interpretation? [duplicate]

I know that the ROC curve plots true positives vs false positives, but does it have any other interpretation, or is it just an arbitrary performance measure? Also, in the case of very unbalanced ...
20
votes
4answers
3k views

What does AUC stand for and what is it?

Searched high and low and have not been able to find out what AUC, as in related to prediction, stands for or means.
1
vote
1answer
174 views

ROC curves for unbalanced datasets

Consider an input matrix $X$ and a binary output $y$. A common way to measure the performance of a classifier is to use ROC curves. In a ROC plot the diagonal is the result that would be obtained ...
0
votes
0answers
22 views

F1 Score vs ROC [duplicate]

How would one decide which performance measure to use when comparing classification algorithms for prediction purposes? Either the F-Score (Precision,Recall) or ROC/AUC analysis or both.
1
vote
1answer
125 views

how to decide which logistic regression model is better?

I have the following 3 models: ...
1
vote
0answers
59 views

When to use the Kappa statistic evaluation metric?

Can someone tell me when is it appropriate to use the Kappa statistic? Also why to use it when one can use Area Under the ROC curve? Or even the Area under the precision-recall curve? So what are the ...
0
votes
0answers
158 views

How to evaluate the optimal cutoff of ROC curve related to logistic regression using roc from the R package pROC?

I would like to get the optimal cutoff of an ROC curve relating to a logistic regression. I am using the roc from the R package pROC. I am assuming same cost of false negative and false positive using ...
0
votes
1answer
37 views

Minimum number of tested patients to have a reasonable ROC curve [closed]

What are the minimum number of tested patients and the acceptable prevalence percent required to have a reasonable ROC curve? for example, can I test a total of 16 patients, 5 are diseased and 11 are ...
4
votes
1answer
80 views

Is up- or down-sampling imbalanced data actually that effective? Why?

I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data. I understand that this could be useful if you're working with a binary (as opposed ...
0
votes
0answers
57 views

Decision function for BernoulliNB classifier. ( for use in plotting ROC and PR curves )

I would like to plot the PR curve using scikit-learn for the Bernoulli Naive Bayes estimator. However, attempting to do so give an error, ...
0
votes
1answer
45 views

Effect of Misclassification Cost on SVM

I am using Matlab to train an SVM for very unbalanced data. However, my concern is not so much for the particular class assignment (ie 1/0), but rather to the scores (the prethreshold continuous SVM ...