Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

Filter by
Sorted by
Tagged with
5
votes
2answers
12k views

How to interpret 95% confidence interval for Area Under Curve of ROC?

(I am following this paper, from page 47 on http://www.bundesbank.de/Redaktion/EN/Downloads/Tasks/Banking_supervision/working_paper_no_14_studies_on_the_validation_of_internal_rating_systems.pdf?...
1
vote
1answer
1k views

Differences between cross validation and bootstrapping to estimate the standard error of the AUC of a given ROC curve

I know there's been some discussion on differences between CV and bootstrapping for estimating out-of-sample prediction error of a classifier. For example, in here (Differences between cross ...
1
vote
1answer
1k views

testing equivalence for two independent AUC

First of all, sorry for the "silly" question. I have two AUC, the first one comes from a training set and the other one comes from a validation set. I am using the ...
5
votes
1answer
5k views

Using partial AUC as Caret metric for cross-validation?

I'm evaluating a grid of tuning parameters using Caret with metric="ROC" for cross-validation. Is there any simple way to use as metric the area under the curve for an specified interval of the ROC ...
1
vote
1answer
717 views

Three way splitting and difference between CV AUC and testing AUC

I have 2000 observations in a dataset with features and a binary-class outcome. I split the dataset into two sets for split sample validation. I use 80% to train the model and internal perform Cross ...
40
votes
2answers
47k views

Area under Precision-Recall Curve (AUC of PR-curve) and Average Precision (AP)

Is Average Precision (AP) the Area under Precision-Recall Curve (AUC of PR-curve) ? EDIT: here is some comment about difference in PR AUC and AP. The AUC is obtained by trapezoidal interpolation ...
1
vote
1answer
6k views

Area under ROC curve for random forest

Does the area under ROC curve depends on which class is defined as default positive class by the random forest model? I am using ...
0
votes
1answer
945 views

Calculating AUC for a GEE

I have used the geeglm package to build a GEE that predicts animal activity (a binary response, active or not) from weather data (e.g., Temperature, a continuous variable). TEMPC <- geeglm(...
0
votes
1answer
1k views

How to calculate AUC for any correlation method?

I want to know how to calculate AUC to compare correlation methods. I read this paper http://www.ncbi.nlm.nih.gov/pubmed/23962479 Is there any idea how the authors of above paper have calculated AUC ...
1
vote
1answer
77 views

AUC per time is nothing else than the mean?

Using AUC is in vogue and has found his place also in clinical research (example). What I don't understand is AUC per time. For example, if a clinical or psychological parameter is measured over time. ...
2
votes
1answer
291 views

So many significant explanatory variables and so small auc

Have you ever seen a model with almost every significant variable and such small auc (area under the ROC curve) ? What might be the cause of it? When I saw summary of a model I thought this model will ...
2
votes
1answer
953 views

Random Forest - What training set measure is the best predictor of test set accuracy?

I'm running a random forest model on a training sample in R in order to make predictions on a hidden test set. I'm having difficulty in understanding how I should go about improving my model in order ...
1
vote
0answers
431 views

Comparing predictors based on ROC AUC and cross-validation error

I am analysing how well some continuous variables (e.g. weight, height) predict the occurrence of a given disease after surgery. I have computed the area under the curve of the receiver-operator ...
93
votes
5answers
131k views

How to calculate Area Under the Curve (AUC), or the c-statistic, by hand

I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model. For example, in the validation dataset, I have the true value for the ...
8
votes
1answer
3k views

Optimizing for AUC

AUC is a popular classification evaluation metric. This is a measure of aggregate performance—do any of the standard loss functions (functions of an individual example's label & prediction) ...
5
votes
1answer
5k views

Equivalent of AUC (area under the ROC curve) for two variables

I was wondering if there is a way to compute AUC using two variables instead of one as predictors. I got two populations after a follow-up, divided in Cases and Controls according to whether they had ...
4
votes
1answer
5k views

How can I calculate AUC using Gini coefficient? [duplicate]

In the Gini Coefficient's Wikipedia page, it is defined as $G= 1 - \frac{\Sigma_{i=1}^n f(y_i)(S_{i-1}+S_i)}{S_n}$ for discrete variables, where $S_i= \Sigma_{j=1}^i f(y_i)y_i$ and $S_0=0$ ($y$ being ...
2
votes
0answers
876 views

Calculating two-tailed p-value from z-score for ROC AUC comparison

I am comparing two predictive models by their bootstrapped ROC AUCs with the method originally described by Hanley and McNeil and modified for bootstrapped data by Robin et al. I'm calculating the z-...
3
votes
1answer
3k views

Training AUC and CV AUC in Boosted Regression Tree

My question is regarding the differences in the training data AUC score and the cross validation AUC score in boosted regression trees (BRT) built using the gbm.step function in the dismo package. I ...
0
votes
1answer
2k views

Is there AUC for neural network?

I am confused about how to calculate AUC for neural network with a softmax classifier. For example, I know that for SVM, we can change the threshold value and determine the AUC. WHat about in neural ...
261
votes
6answers
405k views

What does AUC stand for and what is it?

Searched high and low and have not been able to find out what AUC, as in related to prediction, stands for or means.
2
votes
1answer
2k views

Is it possible to get confusion matrices from AUC?

When I have one confusion matrix for each cutoff level (from 0.00 to 0.99), I can compute AUC coefficient. It looks like: ...
13
votes
1answer
2k views

Connections between $d^\prime$ (d-prime) and AUC (Area Under the ROC Curve); underlying assumptions

In machine learning we may use the area under the ROC curve (often abbreviated AUC, or AUROC) to summarise how well a system can discriminate between two categories. In signal detection theory often ...
-1
votes
1answer
729 views

ROC / AUC for polynomial Labels

How can I calculate the Area Under Curve for a classifier of a polynomial label in Rapidminer? I could only find a performance operator for binomial labels that provides the AUC value.
2
votes
1answer
8k views

R - glmnet - cross validated - AUC [closed]

I have just started working with the glmnet package in R. I have s a dataset which has about 130,000 features and about 32000 rows of data. Here is the code to create the model ...
8
votes
1answer
11k views

What is AUC of PR-curve?

I understand that AUC under ROC curve is a classic evaluation measurement for classifiers (which is basically the accuracy). However, when data is imbalanced, PR will be alternative. So, what does the ...
15
votes
3answers
18k views

What are the differences between AUC and F1-score?

F1-score is the harmonic mean of precision and recall. The y-axis of recall is true positive rate (which is also recall). So, sometime classifiers can have low recall but very high AUC, what that ...
1
vote
1answer
1k views

Significant p value for Mann-Whitney U test but low AUC

How is it possible that for two sample sets I'm getting a low p-value, but also a low AUC value (just below 0.5)? To compute the P-value I'm looking at the second outputted value of the function here ...
2
votes
2answers
3k views

what does it mean when out of sample AUC is greater than in sample AUC?

I am fitting a logistic regression model on a data set with about 200,000 observation and 100 features. According to SAS output, the model converged correctly with an in-sample AUC of 0.85. However, ...
3
votes
1answer
4k views

Differences in AUC calculation in R between pROC and AUC

I was comparing the performance of pROC and AUC libraries when performing auc() calculations on random data: ...
15
votes
3answers
31k views

What is a good AUC for a precision-recall curve?

Because I have a very imbalanced dataset (9% positive outcomes), I decided a precision-recall curve was more appropriate than an ROC curve. I obtained the analogous summary measure of area under the P-...
0
votes
1answer
189 views

Binary input to ROC analysis

Im working on assessment of algorithm sensitivity and specificity. I've developed a simulation in order to detect true and false positives and negatives. My intersest is to know if my algorithm is ...
8
votes
1answer
4k views

Is it reasonable for a classifier to obtain a high AUC and a low MCC? Or the opposite?

Let's say I have 2 models: 1) High Matthew's correlation coefficient (MCC) score, low area under the curve (AUC) 2) Low MCC, high AUC When I say high and low, I mean relatively to the other model. ...
3
votes
1answer
1k views

Differences in AUC calculation between pROC and ROCR

Does anyone know the difference in calculation between these two AUC packages? They get different results when I add in positives with predicted value of 0 (simulating a prob model where many outputs ...
1
vote
1answer
53 views

Is Area under curve a composite function

I have some data examples. If I split the data into three parts and the have some scores for each example of the three parts and then calculate individual AUCs for the three parts In the next case, I ...
4
votes
1answer
2k views

ROC/AUC Confidence Interval

For a single ROC curve (with relevant AUC score), how can you calculate the confidence interval? (The data used to generate this ROC/AUC is available) Given my relatively limited background in this ...
1
vote
1answer
2k views

How can I get cut-off point in multivariated ROC analysis [duplicate]

If I have 1 independent variable (continues) and 1 dependent variable (binary), I can conduct logistic regression and ROC analysis, and I can get a cut-off point of independent variable using ROC ...
3
votes
1answer
4k views

R AUC never less than 0.5?

I'm doing some work with random forests in R using the randomForest package, and I've run into something that seems odd to me. Even when the data is completely ...
3
votes
1answer
9k views

Sample size calculation for ROC/AUC analysis

As a background, I am not familiar with stats except on a basic level. I have been tasked with doing some analysis that is out of my comfort zone. I am trying to figure out how to compute necessary ...
2
votes
1answer
431 views

Statistical Power of ROC/AUC Test with non-IID Samples :: To how many IID Samples are my non-IID Samples Equivalent?

I've been assigned to solve the following problem as part of a serious, biological research project. I think I have a tentative solution, but I'm wondering whether the approach I've picked is the best....
4
votes
3answers
3k views

Can AUC decrease with additional variables?

I'm fitting a logistic regression model to predict probabilities from a set of variables. I'm comparing two such models, say M1 and ...
1
vote
1answer
2k views

Reverse AUC interpretation

Given a classifier (SVM) classifying in 2 'classes' (+1 or -1) for prediction purposes. It has an AUC score of 0.28, meaning its success rate is lower than just random predictions. If I just do the ...
1
vote
0answers
154 views

Comparing AUC vs accuracy [duplicate]

I understand this question has been asked many times however, i am unable to understand the answers well enough and apply to my situation. I have attached 2 screenshots of my model. There are 5 class ...
8
votes
2answers
4k views

What to do AFTER nested cross-validation?

I've searched exhaustively on this forum and elsewhere, and have come across a lot of great material. However, I'm ultimately still confused. Here's a basic, concrete example of what I'd like to ...
5
votes
4answers
396 views

The value of adding the ROC graph if the AUC is given

I always see in papers that when they want to show how they classifiers performed, they provide ROC graph and the AUC score. Now as far as I know only the AUC tells how well the classifier performed, ...
17
votes
3answers
15k views

Area under the ROC curve or area under the PR curve for imbalanced data?

I have some doubts about which performance measure to use, area under the ROC curve (TPR as a function of FPR) or area under the precision-recall curve (precision as a function of recall). My data is ...
32
votes
3answers
34k views

Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

I have two classifiers A: naive Bayesian network B: tree (singly-connected) Bayesian network In terms of accuracy and other measures, A performs comparatively worse than B. However, when I use the R ...
7
votes
1answer
3k views

Statistics for Area under the ROC curve

I have a question regarding statistical evaluation of the AUC. In their paper (http://www.jstor.org/stable/2531595), DeLong et al. describe a method to evaluate AUC curves. (Another good explanation ...
2
votes
2answers
421 views

Predicting class probabilities in regression based on area under the curve

Logistic regression models the log odds. That is for rv $Y$ which is binary logit$(Y=1)=X\beta$. Then with this model, you can estimate the class probabilities and hence prediction or ...
2
votes
1answer
5k views

Comparison of two logistic regression models (significant result with anova() but very similar AUCs)

I have compared two logistic regression models using the function anova(mod1,mod2,test="Chisq") in R. The result that I obtained is the following: ...

1
6 7 8
9
10