Tagged Questions

Receiver Operating Characteristic, also known as ROC curve.

learn more… | top users | synonyms

2
votes
0answers
24 views

Connections between d' (d-prime) and AUC (Area Under the ROC Curve); underlying assumptions

In machine learning we may use the area under the ROC curve (often abbreviated AUC, or AUROC) to summarise how well a system can discriminate between two categories. In signal detection theory often ...
1
vote
1answer
36 views

Testing Logistic Regression Classifier in R

I am testing the logistic regression classifier in R. I created some test data like this: x=runif(10000) y=runif(10000) df=data.frame(x,y,as.factor(x-y>0)) ...
0
votes
1answer
15 views

ROC / AUC for polynominal Labels

How can I calculate the Area Under Curve for a classifier of a plynominal label in Rapidminer? I could only find a performance operator for binominal labels that provides the AUC value.
0
votes
0answers
15 views

multiple ROC curves in one image (Bayes predictor) [closed]

I have several Bayes models made in KNIME and I need to plot their ROC curves in one image. In KNIME there is ROC curve node, but it can't plot more than one ROC curves at one time (or I don't know ...
0
votes
0answers
27 views

How do we generate the ROC curve for Linear Discriminant Analysis method

I know the method to generate the ROC curve for other methods such as naive Bayes where the tuning parameter is the threshold like also in logistic regression. If we want to generate the ROC curve ...
1
vote
1answer
56 views

Better in ROC AUC vs. better in PR AUC

I'm comparing two classification models by computing the area under ROC and Precision-Recall curves. However sometimes one model is better with AU-ROC but worse in AU-PR, and other times it's better ...
0
votes
1answer
24 views

What is AUC of PR-curve?

I understand that AUC under ROC curve is a classic evaluation measurement for classifiers (which is basically the accuracy). However, when data is imbalanced, PR will be alternative. So, what does the ...
1
vote
2answers
61 views

Help with understanding statistical measures and Receiver Operating Characteristics

In my machine learning class we just went over statistical measures and plots. We looked at the definitions of True Positive Rate (sensitivity/recall, etc), 1-False Positive Rate (speciicity), ...
1
vote
1answer
42 views

Why AUC-PR increases when the number of positives increase?

I asked a question earlier about comparing models using Precision-Recall AUC. One of the answers included the following statement: "The larger the fraction of positives in the data set, the larger the ...
1
vote
1answer
41 views

Ranking two models based on ROC-AUC and PR-AUC

I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed ...
2
votes
1answer
33 views

Is it correct to use Precision-Recall AUC in a balanced dataset situation?

I have a binary classification scenario with a dataset that is unbalanced (much more negatives than positives). When I train a classifier on this dataset I get a Precision-Recall AUC of 0.7. Then I ...
0
votes
0answers
25 views

How to calculate the area under the precision-recall curve for the random classifier?

I know that the random classifier score in ROC AUC (Area under the curve) is always 0.5. My question is: how to calculate the Area under the precision-recall curve for the random classifier?
0
votes
1answer
52 views

What is the convex hull in ROC curve?

I'm reading a paper about ROC and PR curves. They mentioned the ROC convex hull but they don't define it or say what it is. Can someone please tell me the meaning of it? What is a convex hull in ROC ...
2
votes
2answers
55 views

An intuitive meaning of the area under the PR curve?

Wikipedia says that an interpretation of the area under the ROC curve is: "the area under the curve is equal to the probability that a classifier will rank a randomly chosen positive instance higher ...
1
vote
1answer
116 views

Finding true positive / negative and false positive / negative rates using R

I have a data frame with two classes. I want to find the true positive and false positive rate and then plot the ROC curve. I tried this: ...
0
votes
0answers
8 views

Tree classifier or nested model?

I am new to statistics and was wondering what the right kind of model to use for the following scenario. I have two sets of continuous observations A and B for 50 samples. These 50 samples are ...
0
votes
2answers
40 views

How to turn my data into a ROC curve in R? [closed]

I'd like to generate ROC curves in R, but I'm confused about what input to give either ROCR or pROC. I have 5000 data-points for which I know the true classification (1 or 0), and a continuous ...
0
votes
0answers
50 views

Complete Logistic Regression framework using K-Cross validation

I'm implementing a logistic regression model in a low event rate data. I have gone through many webpages (including stackoverflow, including my questions) but none answer or describe the end-to-end ...
0
votes
0answers
45 views

Significance of a dichotomized variable from a continuous variable

I am analyzing a X continuous independent variable with a Y binary response. The investigator has interested on dichotomize the X variable by the “best” cutpoint from the ROC curve (for example the ...
0
votes
2answers
71 views

ROC (Epi library) how to calculate TP, FN,TN, FP

I'm trying to find how to compute the true negative (TN), false negative (FN), true positive (TP), and false positive (FP) if I have a cutpoint like in the following picture: ...
0
votes
2answers
32 views

Use of data from ROC curve

In order to find an optimal time for initiation of treatment post surgery (oncologic patients) I created a ROC curve with death defined as event. The AUC was not significant. However, I decided to use ...
0
votes
0answers
29 views

Is it logical to stand on the chance-line (50-50 %) when we don't know a-priori probabilities?

Let we have two hypotheses $H_0$ and $H_1$ and we don't know their a-priori probabilities. If we wish to calculate the average probability of error, does it makes sense to assume 50-50 % chance of ...
0
votes
0answers
37 views

How to evaluate a trained model using parameters other than AUC in RapidMiner?

I am using RapidMiner to build predictive models trained and cross-validated by a set of medical data(65 cases. 18 attributes), I am now running trials by trying different combinations of learners and ...
0
votes
0answers
52 views

Cutoff and precision values of a binary classifier

Let's say I have fitted a binary classifier to some data and I'm varying the cutoff value, effectively producing a ROC-curve. Knowing the true proportions of positives and negatives, I can calculate ...
2
votes
2answers
94 views

ROC for more than 2 outcome categories

How do you construct ROC Curves when there are more than two outcome categories (in my case, I have four)? I've heard you should do this for the most popular group. Are there any other ideas? Are ...
0
votes
0answers
37 views

Selecting individuals from a population using a binary classifier

I have a dataset consisting of around 200 individuals, whose outcome is either of state $0$ or $1$. I am able to make binary classifiers and predictors on this set and build ROC-curves for them just ...
1
vote
3answers
128 views

ROC graph shape

Could you explain to me how the shape of a ROC curve is determined? From the following illustration, it seems that for every time the actual class (C) is positive, it goes up and when it's negative, ...
2
votes
0answers
103 views

Unbalanced dataset - ROC curve to compare classifiers?

I use the machine learning software WEKA for data mining on biological data. I would describe my dataset as unbalanced: It comprises around 2000 instances, ...
2
votes
1answer
99 views

Is it reasonable for a classifier to obtain a high AUC and a low MCC? Or the opposite?

Let's say I have 2 models: 1) High Matthew's correlation coefficient (MCC) score, low area under the curve (AUC) 2) Low MCC, high AUC When I say high and low, I mean relatively to the other model. ...
-1
votes
1answer
27 views

Which is better ROC curve or Confusion matrix?

anyone know about the predictive model evaluation? I'm confused about the ROC curve and confusion matrix. The area under the curve for ROC is represent about the accuracy of the classifier. But what ...
1
vote
2answers
311 views

Using the caret package is it possible to obtain confusion matrices for specific threshold values?

I've obtained a logistic regression model (via train) for a binary response, and I've obtained the logistic confusion matrix via ...
0
votes
0answers
57 views

Multi-class AUC in Matlab

I would like to compute the area under the ROC-courve (AUC) metric for a classifier with multiple classes. Do you know (reliable) functions for Matlab that implement methods for that, like e.g. in ...
2
votes
1answer
95 views

Differences in AUC calculation between pROC and ROCR

Does anyone know the difference in calculation between these two AUC packages? They get different results when I add in positives with predicted value of 0 (simulating a prob model where many outputs ...
3
votes
1answer
125 views

ROC-AUC and Precision-Recall for random classifiers in class imbalanced problems

I have always always understood the diagonal of the ROC plot to represent the performance of a "random" classifier (corresponding to an AUC of 0.5). Is this still the case for highly imbalanced ...
2
votes
2answers
212 views

ROC vs. Accuracy [duplicate]

If you want to compare two learning algorithms, which metric is better to use in general: ROC or accuracy? I understand that in ROC, you get both the sensitivity and specificity?
1
vote
1answer
49 views

ROC/AUC Confidence Interval

For a single ROC curve (with relevant AUC score), how can you calculate the confidence interval? (The data used to generate this ROC/AUC is available) Given my relatively limited background in this ...
0
votes
1answer
43 views

How can I get cut-off point in multivariated ROC analysis

If I have 1 independent variable (continues) and 1 dependent variable (binary), I can conduct logistic regression and ROC analysis, and I can get a cut-off point of independent variable using ROC ...
2
votes
1answer
268 views

Sample size calculation for ROC/AUC analysis

As a background, I am not familiar with stats except on a basic level. I have been tasked with doing some analysis that is out of my comfort zone. I am trying to figure out how to compute necessary ...
0
votes
0answers
23 views

Computing baseline probability

What is the meaning of the term Baseline probability in an experiment? How is it computed, say for a binary classifier? How to measure the performance of a classifier according to a given Baseline ...
2
votes
2answers
425 views

How we can draw an ROC curve for decision trees?

Normally we cannot draw an ROC curve for the discrete classifiers like decision trees. Am I right? Is there any way to draw an ROC curve for Dtrees?
2
votes
1answer
277 views

Understanding ROC curve[edited]

I'm having trouble understanding ROC curve. Is there any advantage/improvement in area under the ROC curve if I build different models from each unique subset of train set and use it to produce ...
0
votes
1answer
76 views

Confidence intervals for predictors in multivariate logistic regression

I've got a question. I am dealing with medical data which contain 5 predictors and 1 binary outcome. When I try to classify the data using all 5 predictors I get 0.84 area under roc-curve which is ...
0
votes
0answers
112 views

recursive feature elemination in R with caret

i work with R caret software package to select the most important features from some set of data. My response is a factor of multiple classes (e.g. nominal ...
0
votes
1answer
75 views

Confused with ROC curve and interpretation

The following figures show examples of ROC curves: First of all ignoring the picture, from a logical point one can say: When the cutoff value decreases, more and more cases are allocated to class 1 ...
0
votes
0answers
40 views

ROC and false positive rate with over sampling

I'm modelling a rare event (say 1 in 10000) and I'm using an over sampled train set to cross validate and train my model. I'm using ROC as a global performance metric but there are business reasons ...
1
vote
0answers
115 views

Comparing ROC-curves

I would like to find if there is a significant difference between two ROC-curves. I've found the roc.test in the pROC package. However, I cannot seem to find any information on how this test is ...
1
vote
1answer
60 views

Reverse AUC interpretation

Given a classifier (SVM) classifying in 2 'classes' (+1 or -1) for prediction purposes. It has an AUC score of 0.28, meaning its success rate is lower than just random predictions. If I just do the ...
0
votes
1answer
78 views

Problem with ROC curve in R

I am trying to plot the ROC curve for a random forest model (ROCR package), and I am getting weird results. I have double-checked the code several times, but I ...
1
vote
0answers
37 views

How to visualize the effect of a regression parameter

I am arguing that I can control error vs. coverage by modifying a certain parameter. After running an experiment with leave-many-out validation I have a set of errors along with the parameter value ...
4
votes
0answers
205 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...