Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

Filter by
Sorted by
Tagged with
1
vote
1answer
254 views

Stacking AUC vs. average over folds

I have a two class prediction problem where in one class I have 70% of the samples and in the other class 30% of the samples, so class imbalance. I'm conducting 10-fold cross-validation. To calcualte ...
2
votes
0answers
38 views

Comparing AUC between two subsets of test-fold

I suspect that the out-of-sample AUC of my model depends on the number of events in the out-of-sample/testing fold (I am modelling a binary variable, 0/1). In order to test this hypothesis, I want to ...
1
vote
0answers
136 views

Aggregating ROC AUC values of several Logistic Regression Models

I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
2
votes
1answer
163 views

Area Under the Curve - Variable and Log Transformed Variable

I have a situation where I am fitting two simple logistic regression models - one model with the variable of interest included as the only predictor, and the other model with the log of the variable ...
1
vote
0answers
98 views

What is a reliable way to obtain an optimism-correct AUC with confidence limits?

I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC). I am trying to look at a method with ...
0
votes
1answer
93 views

Is it correctly understood that ROC/AUC cannot be calculated without flexible criterion value?

I have a proprietary predictor that simply gives me a binary output. Let's say that it is detecting faulty units. In a set where 27 units are faulty and 76 units are working the predictor correctly ...
0
votes
1answer
50 views

Minimising CV standard deviation to increase accuracy

Does minimising the standard deviation of CV folds have any correlation with model accuracy as a theme? I've noticed that changing the order of rows in a training data can change the AUC for each CV ...
2
votes
2answers
310 views

Why does pROC roc work with non-probability predictions?

With the pROC package, I can do this: true <- c(1, 1, 1, 0) predicted <- c(0.5, 0.1, 0.6, 0.1) roc(true, predicted) which gives as expected: ...
0
votes
1answer
194 views

High AUC and Accuracy but weird output in confusion matrix

I am working on image classification problem to determine gender given a face. The dataset is located here gender face dataset on kaggle (link to my notebook). The class distribution is as follows. <...
0
votes
1answer
211 views

Why there is decrease in AUC values with increase in number of iteration in Held out validation?

I have a dataset with 600 rows and 4000 columns for which I am trying to do held-out cross-validation with 10 and 100 iterations. At first, the dataset is split into 80%:20% training and “held-out” ...
2
votes
1answer
1k views

Why is cross_val_score substantially lower than .score or roc_auc_score?

I have a trained model, a GradientBoostingClassifier. My dataset is 60 thousand something rows of data that I've split into 66/33 train/test sets. Scoring the model via the ...
1
vote
1answer
42 views

Relationship between accuracy and number of independent variable in Logistic Regression

I am working on classification problem https://www.kaggle.com/henriqueyamahata/bank-marketing I used SAS and proc logistic :: stepwise selection(sle=0.05,sls=0.05) procedure to reduce the variable ...
1
vote
0answers
48 views

Sample size for AUC based on mean and SD of raw data

I'm trying to estimate the sample size required to achieve 80% power at a 5% significance level for a superiority study comparing AUC from bioavailability data. My challenge is the only previous ...
1
vote
1answer
3k views

How to distinguish overfitting and underfitting from the ROC AUC curve?

For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model. But how to distinguish whether a ...
2
votes
1answer
120 views

K-NN optimal value of 'K'

How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score? Do I need ...
1
vote
1answer
60 views

AUC ROC when one class consists of smaller subclasses

This question is different from Binary classification when one class consists of multiple subclasses I have two classes that I want to distinguish by a supervised learning classifier such as a random ...
2
votes
1answer
1k views

Does AUC/ROC curve return a p-value?

When reading this article, I noticed that the legend in Figure 3 gives a p-value for each AUC (Area Under the Curve) from the ROC (Receiver Operator Characteristic) curves. It says: The area under ...
4
votes
2answers
631 views

Appropriate way to get Cross Validated AUC

I was thinking about cross-validation and how it is the most appropriate way to do it... Let's take the case of binary logistic regression where the goal is to calculate the AUC. Make the partition ...
1
vote
0answers
110 views

What is the integral of the False Positive Rate over the False Positive Rate, compared to the AUC?

In machine learning the Area Under the Receiver Operating Characteristic Curve ($AUC$) can be illustrated in a plot of the True Positive Rate ($TPR$) against the False Positive Rate ($FPR$). Formally, ...
2
votes
0answers
77 views

How to calculate the standard error of ratio of two AUC

Please can anyone help me with this! I hope my explanation is going to be simple: This is the data I have: I only have access to the aggregated data as below. the first column is the time points a ...
2
votes
1answer
898 views

How do I calculate AUC with leave-one-out CV

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
2
votes
2answers
966 views

Can I trust my random forest model with low sample size and unequal class distribution?

I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution. I am doing some explorative modeling by using 400 features to classify ...
1
vote
0answers
110 views

hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R

I would like to calculate the variance of the AUC of readers (for each reader and averaged results) giving a score(1-5) to ...
3
votes
1answer
2k views

Reason for higher AUC from a test set than a training set using a random forest

I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced ...
1
vote
1answer
62 views

Why are ROC curves and AUC values not always relevant?

So I have read in posts and in literature (Frank Harrel - Regression Modeling Strategies fx) that depending on what you do, ROC curves and AUC values are not always relevant, but often written in ...
1
vote
1answer
58 views

ROC curve: models have different FPR ranges

I've created ROC curves by calculating the TPR and FPR at various thresholds. The FPR range differs between models, so I'm wondering if AUC is still a valid way to compare the curves. A curve will ...
3
votes
1answer
353 views

How is iso accuracy line related to ROC curve

I have read many articles about ROC curve. Some specified a method to calculate the accuracy of a classifier using iso accuracy lines in a convex hull. Some article examples: http://mlwiki.org/index....
3
votes
2answers
745 views

Why AUC is not a good performance metric for a classification model?

After understanding the benefits of AUC I was stumbled to know that in some scenarios it might not be a good performance metric for evaluating a classification model. The below are the 2 scenarios: ...
2
votes
1answer
91 views

how to interpret the behavior of a ROC curve very close to one

annex the ROC curve of a logit model, which has a very high AUC, how can I interpret the behavior of this curve, I doubt the logit model?
0
votes
0answers
62 views

AUC and accuracy interpretation

An accuracy $80\%$ of a model that predicts binary outcomes is interpreted as: Given a sample whose outcomes we want to predict, 80% of the prediction will be correct. What does an AUC of $80\%$...
2
votes
1answer
81 views

Why are the trivial points included when calculating AUC?

I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue ...
7
votes
2answers
11k views

What is the formula to calculate the area under the ROC curve from a contingency table?

For example, if my table is: ...
2
votes
1answer
820 views

How to Interpret AUROC score?

My model has an AUROC value of 0.7, and I have a 75:25 class (75% negative, 25% positive) imbalance. From my understanding, AUROC is calculated by using different thresholds for considering the ...
1
vote
2answers
556 views

Do you need to calculate sample size to evaluate a new diagnostic test?

I am writing a grant application which will be evaluating a new diagnostic test. The test will predict whether a patient with lung fibrosis will remain stable or progress. I am using an existing ...
0
votes
2answers
45 views

Performance evaluation

I'd like to test the performance of a penalized regression. I did three separate regressions for each response variable (one numerical, one binomial and one multinomial). I was checking this link, and ...
3
votes
0answers
49 views

Confidence intervals of AUC obtained by merging/pooling predictions from different test sets

I have one question regarding the CIs of the AUROC calculated merging/pooling the predictions coming from different test sets. In one analysis, we use a sort of nested cross-validation approach, ...
1
vote
0answers
37 views

How to make really bad results from a machine learning model better by reversing predictions

I trained a classification model on some data with two classes and have really low accuracy. I have a false-positive rate of 86 % for both classes I am trying to predict. I was wondering if I could ...
1
vote
0answers
39 views

How to determine the optimal threshold for my classification problem using fpr ,tpr values for each classification record?

The output of my prediction using classification algorithm is in dataframe that contains TPR and FPR value for each prediction. Suppose I have 100 records for prediction then in that cade my data ...
0
votes
0answers
105 views

Density curve in R - AUC bigger than 1

correct me if I'm wrong but I was expecting the area under the curve should be 1 for a probability density function. Can anybody explain why it's not always the case when using the ...
1
vote
0answers
647 views

Micro- or macro-averaged AUC for highly imbalanced data?

I have a classification problem with 3 classes. With random forest classifier I'm getting the following confusion matrix: The micro-averaged AUC is 0.76 and the macro-averaged AUC is 0.55. On the ...
1
vote
0answers
175 views

How to interpret ROC curve with confusion matrix and F1 score?

I have implemented a random forest classifier to do a binary classification in highly imbalanced class. As the performance measurements, ROC and the f1 score was considered. However, the ROC curve ...
1
vote
1answer
411 views

Is it possible for a model to have higher sensitivity/specificity but lower accuracy and AUC?

In the evaluation of classification models, I've found one model to have a higher accuracy and c-statistic (AUC) as compared to a second model. However, the second model has higher sensitivity, ...
1
vote
0answers
54 views

How to derive the the AUROC from the Bayes Minimum Risk (Hand 2009)?

The area-under-the-receiver-operating-characteristic-curve (AUROC) can be derived from the Bayes Minimum Risk. The derivation requires the assumption that the exact costs are unknown but follow a ...
1
vote
0answers
82 views

Poor P-R curve for binary classifier trained on balanced data, with imbalanced test data

I have a very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing). I am performing classification using ...
1
vote
3answers
2k views

When is an AUC score misleadingly high?

I have an algorithm which gives an AUC (area under the receiver operating curve) of 0.94. I mean, this is amazing, but... probably too amazing, considering the difficulty of the task I am working on. ...
14
votes
3answers
3k views

Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity

I have a test set of 100 cases and two classifiers. I generated predictions and computed ROC AUC, sensitivity and specificity for both classifiers. Question 1: How can I compute p-value to check if ...
2
votes
1answer
741 views

Improve the precision of random forest for count data

I am trying to create a classification model that predicts whether a customer will enquire for a financial product based on some 250 independent variables. 98% of the variables are count variables and ...
0
votes
2answers
623 views

Possible to optimize for area under the precision-recall curve in glmnet logistic regression?

tl;dr with the R glmnet package, is it possible to optimize for the area under the precision-recall curve, rather than the area ...
0
votes
0answers
99 views

Comparing AUCs of ROC of same diagnostic test on different samples

Supposed I've got a sample of 200 subjects, and based on these subjects, I determined the AUC of the ROC of a diagnostic test. Next, from these 200 subjects, I drew a subsample of the diagnostic test'...
1
vote
0answers
75 views

A better way to compare accuracy?

Hi I have an algorithm that takes a single sample, call it i and tries to predict what other samples in a cohort it is most closely related to. This cohort consist of N=11K from different tissues. ...

1 2
3
4 5
9