Receiver Operating Characteristic, also known as ROC curve.

learn more… | top users | synonyms

0
votes
0answers
5 views

ROC Analyses - can collapse assessment time periods?

I hope you can provide some advice on doing ROC analyses. I have been running ROC analyses on several different screening tests and have a question about one of my data sets. It is a longitudinal data ...
2
votes
1answer
20 views

ROC and constant factor on probabilities

I play around with a few data to learn and I am wondering about something; I can evaluate my results with ROC which is processed from FP and FN. I had predicted a few probabilities for my events to ...
0
votes
0answers
51 views

ROC curve cut off and weights

I have a dependent variable distinguishing between patients that should go to treatment A or treatment B. I want to develop a questionnaire containing binary variables that should decide if the ...
0
votes
0answers
8 views

Is there any lower limit for number of positives when generating lift plot?

I am wondering if there is any condition on number of positives in test set when I am trying to compute lift plot to check the properties of my classifier?
9
votes
2answers
413 views

Difference between regression analysis and curve fitting

Can anybody please explain to me the real difference(s) between regression analysis and curve fitting (linear and nonlinear), with an example if possible? It seems that both try to find a ...
0
votes
0answers
27 views

Interpretation and meaningfulness of regression coefficients

I have performed logistic regression on banking data which is trying to predict the bad customers correctly due to the cost involved. I have build a model and pasting a picture of the output obtained ...
0
votes
0answers
10 views

ROC and post estimation COX Harrell's C using your dataset

I have built a predictive model using a combination of logistic and cox regression models. I did it using a dataset of about 5000 records. I would like to calculate the AUC and the Harrell's post ...
2
votes
1answer
38 views

So many significant explanatory variables and so small auc

Have you ever seen a model with almost every significant variable and such small auc (area under the ROC curve) ? What might be the cause of it? When I saw summary of a model I thought this model will ...
0
votes
0answers
27 views

How is the mean area under the curve calculated?

I am using 10-fold cross-validation for performance estimation. From each of the ten iterations, I get an area under the curve (AUC) metric, e.g. ...
0
votes
0answers
38 views

Area under ROC curve vs. Accuracy in unbalanced sample

I have a binary classification problem with 3000 samples (number of 1 as outputs = 300, number of 0 as outputs = 1700). After balancing database (selecting 300 samples from 0 outputs) I trained the ...
0
votes
0answers
24 views

Constructing an ROC curve without true negatives?

I’m not really a statistician by training so please excuse any incorrect terminology or if this is a painfully obvious question. I am trying to assess the performance of two algorithms. I don’t wish ...
3
votes
1answer
62 views

Simple Question about ROC Curve

Motivated by this reference, it states under ROC Space When evaluating a binary classifier, we often use a Confusion Matrix...however here we need only TPR and FPR I'd feel more comfortable if ...
4
votes
1answer
142 views

Different shapes of an ROC curve

What are the possible shapes of an ROC curve? Is it necessary for an ROC curve to be shaped like a normal distribution curve? Can we regard the following two curves as ROC with the area under the ...
0
votes
0answers
17 views

Forward sequential feature selection improving classifier performance?

I was in a bit of a conversation with a co-worker about using forward selection. My training data is on order of ~6,000 w/ dimensionality of 1,200, and testing data on order of ~3,000. Currently, I'm ...
3
votes
1answer
56 views

Getting the optimal age cut off

I am working with a database of diseased and non-diseased patients. I would like to recommend an optimal age cut off for disease screening. The common problem with screening is that it creates a ...
0
votes
0answers
26 views

ROC for optimal cut-off using Svyglm object

I am trying to estimate optimal cut-points via ROC for a complex survey data. I able to achieve this task with ROC(in Epi) and OptimalCutpoints package but for the unweighted sample. The ...
0
votes
1answer
59 views

Model comparison between glm (with Firth correction), random Forest, penalised SVM

I am currently developing three models to classify features of gene sites. I was using glm (with Firth correction), random Forest and SVM to build the models and I used forward and backward ...
1
vote
2answers
52 views

Using ROC curve for balanced data

I understand that using the area under the ROC curve is a useful error measurement for unbalanced data. What happens if we use it for balanced data?
1
vote
0answers
73 views

How can I calculate AUC using Gini coefficient?

In the Gini Coefficient's Wikipedia page, it is defined as $G= 1 - \frac{\Sigma_{i=1}^n f(y_i)(S_{i-1}+S_i)}{S_n}$ for discrete variables, where $S_i= \Sigma_{j=1}^i f(y_i)y_i$ and $S_0=0$ ($y$ being ...
1
vote
0answers
84 views

Calculating two-tailed p-value from z-score for ROC AUC comparison

I am comparing two predictive models by their bootstrapped ROC AUCs with the method originally described by Hanley and McNeil and modified for bootstrapped data by Robin et al. I'm calculating the ...
1
vote
1answer
33 views

ROC curve when the numbers of both total positives and negatives are unknown

I have to use a roc curve to find an agreement between sensitivity between the FPR and TPR. The problem is that I only know the true positives. How could I circumvent this? More accurate, I have a ...
0
votes
1answer
42 views

How to know if my data is balanced or imbalanced for an ROC curve analysis?

I am doing a research on the reliability of different models in detecting hidden defects in a test specimen. I have made a test specimen with defect prevalence about 25% (12 positive out of 49 total ...
0
votes
1answer
34 views

What does ROC-EER in percent stand for?

Ive tried to understand what the ROC Curve represents and what EER (Equal Error Rate) means. And I somehow think I got to understand some of the explanations I read on the internet and videos I ...
6
votes
2answers
535 views

d prime with 100% hit rate probability and 0% false alarm probability

I would like to calculate d prime for a memory task that involves detecting old and new items. The problem I have is that some of the subjects have hit rate of 1 and/or false alarm rate of 0, which ...
3
votes
1answer
126 views

One model performs better than the other. How to measure if it is statistically significant?

So, let's say that I train two models on the same dataset. I run the experiment once and I get the following results: Using a Neural Network I get an AUC ROC of 0.941. Using Random Forest I get an ...
4
votes
1answer
114 views

Is the objective to beat a random classifier when the data set is skewed using PR curves?

I have a testing data set where 1/3 of the observations are class-1 objects and the remainder class-0. Hence, the data set is skewed (skewed classifier), literature suggests that if the data set is ...
1
vote
1answer
90 views

Threshold selection by intersection of Sensitivity and Specificity

Some days ago, I learned in a lecture that the intersection of Sensitivity and Specificity provides an optimal compromise for choosing a classification threshold for logit or probit models. However, ...
0
votes
0answers
23 views

Recommended performance metrics of a binary classifier having an example-based cost matrix

I would like to know what are the recommended performance metrics to assess a binary classifier, when the cost matrix is changing for each sample. My problem comes from the fact that in this case, ...
2
votes
0answers
29 views

does the area under the curve (AUC) has some interpretation? [duplicate]

I know that the ROC curve plots true positives vs false positives, but does it have any other interpretation, or is it just an arbitrary performance measure? Also, in the case of very unbalanced ...
18
votes
4answers
2k views

What does AUC stand for and what is it?

Searched high and low and have not been able to find out what AUC, as in related to prediction, stands for or means.
1
vote
1answer
103 views

ROC curves for unbalanced datasets

Consider an input matrix $X$ and a binary output $y$. A common way to measure the performance of a classifier is to use ROC curves. In a ROC plot the diagonal is the result that would be obtained ...
0
votes
0answers
19 views

F1 Score vs ROC [duplicate]

How would one decide which performance measure to use when comparing classification algorithms for prediction purposes? Either the F-Score (Precision,Recall) or ROC/AUC analysis or both.
1
vote
1answer
116 views

how to decide which logistic regression model is better?

I have the following 3 models: ...
1
vote
0answers
47 views

When to use the Kappa statistic evaluation metric?

Can someone tell me when is it appropriate to use the Kappa statistic? Also why to use it when one can use Area Under the ROC curve? Or even the Area under the precision-recall curve? So what are the ...
0
votes
0answers
130 views

How to evaluate the optimal cutoff of ROC curve related to logistic regression using roc from the R package pROC?

I would like to get the optimal cutoff of an ROC curve relating to a logistic regression. I am using the roc from the R package pROC. I am assuming same cost of false negative and false positive using ...
0
votes
1answer
30 views

Minimum number of tested patients to have a reasonable ROC curve [closed]

What are the minimum number of tested patients and the acceptable prevalence percent required to have a reasonable ROC curve? for example, can I test a total of 16 patients, 5 are diseased and 11 are ...
4
votes
1answer
69 views

Is up- or down-sampling imbalanced data actually that effective? Why?

I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data. I understand that this could be useful if you're working with a binary (as opposed ...
0
votes
0answers
49 views

Decision function for BernoulliNB classifier. ( for use in plotting ROC and PR curves )

I would like to plot the PR curve using scikit-learn for the Bernoulli Naive Bayes estimator. However, attempting to do so give an error, ...
0
votes
1answer
37 views

Effect of Misclassification Cost on SVM

I am using Matlab to train an SVM for very unbalanced data. However, my concern is not so much for the particular class assignment (ie 1/0), but rather to the scores (the prethreshold continuous SVM ...
2
votes
3answers
494 views

How to interpret a ROC curve?

I applied logistic regression to my data on SAS and here are the ROC curve and classification table. I am comfortable with the figures in the classification table, but not exactly sure what the roc ...
2
votes
3answers
144 views

Is there any other measure of the performance of a classifier than the area under the ROC curve?

I am trying to draw an ROC curve for a classifier and wondered to know if there is any other measure for the performance of the classifier than the AUC. And is there any free software that I can use ...
3
votes
0answers
105 views

Connections between d' (d-prime) and AUC (Area Under the ROC Curve); underlying assumptions

In machine learning we may use the area under the ROC curve (often abbreviated AUC, or AUROC) to summarise how well a system can discriminate between two categories. In signal detection theory often ...
1
vote
1answer
100 views

Testing Logistic Regression Classifier in R

I am testing the logistic regression classifier in R. I created some test data like this: x=runif(10000) y=runif(10000) df=data.frame(x,y,as.factor(x-y>0)) ...
0
votes
1answer
73 views

ROC / AUC for polynominal Labels

How can I calculate the Area Under Curve for a classifier of a plynominal label in Rapidminer? I could only find a performance operator for binominal labels that provides the AUC value.
0
votes
0answers
246 views

How do we generate the ROC curve for Linear Discriminant Analysis method

I know the method to generate the ROC curve for other methods such as naive Bayes where the tuning parameter is the threshold like also in logistic regression. If we want to generate the ROC curve ...
2
votes
2answers
142 views

Better in ROC AUC vs. better in PR AUC

I'm comparing two classification models by computing the area under ROC and Precision-Recall curves. However sometimes one model is better with AU-ROC but worse in AU-PR, and other times it's better ...
0
votes
1answer
80 views

What is AUC of PR-curve?

I understand that AUC under ROC curve is a classic evaluation measurement for classifiers (which is basically the accuracy). However, when data is imbalanced, PR will be alternative. So, what does the ...
1
vote
2answers
98 views

Help with understanding statistical measures and Receiver Operating Characteristics

In my machine learning class we just went over statistical measures and plots. We looked at the definitions of True Positive Rate (sensitivity/recall, etc), 1-False Positive Rate (speciicity), ...
1
vote
1answer
107 views

Why AUC-PR increases when the number of positives increase?

I asked a question earlier about comparing models using Precision-Recall AUC. One of the answers included the following statement: "The larger the fraction of positives in the data set, the larger the ...
1
vote
1answer
99 views

Ranking two models based on ROC-AUC and PR-AUC

I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed ...