Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
0
votes
0answers
15 views
hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R
I would like to calculate the variance of the AUC of readers (for each reader and averaged results) giving a score(1-5) to ...
0
votes
0answers
24 views
Class distribution reflected in AUC?
I'm learning on churn models and i found a curious result. I tried a logistic regression and random forest model and used k-fold and ROC metric. I created a model for each group of customer (20). ...
0
votes
0answers
18 views
Reason for higher AUC from a test set than a training set using a random forest
I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced ...
1
vote
1answer
22 views
Why are ROC curves and AUC values not always relevant?
So I have read in posts and in literature (Frank Harrel - Regression Modeling
Strategies fx) that depending on what you do, ROC curves and AUC values are not always relevant, but often written in ...
1
vote
1answer
19 views
ROC curve: models have different FPR ranges
I've created ROC curves by calculating the TPR and FPR at various thresholds. The FPR range differs between models, so I'm wondering if AUC is still a valid way to compare the curves. A curve will ...
1
vote
1answer
29 views
How do iso accuracy line related to ROC curve
I have read many articles about ROC curve. Some specified a method to calculate the accuracy of a classifier using iso accuracy lines in a convex hull.
Just like in this articles :
http://mlwiki.org/...
0
votes
0answers
8 views
Different outcomes of the classification algorithms with similar AUC score
I am new to applied statistics and have a question to some more experience folks regarding the outcome of the models I've built. My data is highly imbalanced and I use oversampling to even out the ...
0
votes
0answers
44 views
Sample size calculations for comparing area under curve (AUC) in ROC analysis in R
I have a dataset with a response variable and a predictor variable. I want to calculate the possible difference of AUC (delta) I can find with power = 0.8 and significance level = 0.05. Let's say I ...
2
votes
2answers
81 views
Why AUC is not a good performance metric for a classification model?
After understanding the benefits of AUC
I was stumbled to know that in some scenarios it might not be a good performance metric for evaluating a classification model. The below are the 2 scenarios:
...
0
votes
0answers
29 views
Why is my ROC curve above the random line but the AUC is very low?
A little bit of background on my development process. I'm using a Random Forest model using the software package Alteryx (R Based) to classify a binary target variable that is approximately 60 (Neg - ...
2
votes
1answer
31 views
how to interpret the behavior of a ROC curve very close to one
annex the ROC curve of a logit model, which has a very high AUC, how can I interpret the behavior of this curve, I doubt the logit model?
0
votes
0answers
5 views
Compare the performance of a new model with the existing model
The challenge for me here is that I do not know the test/train datasets of the EXISTING model but I have the model with me (it is a logistic regression equation).
I believe the NEW model's AUC on ...
0
votes
0answers
24 views
Compare two binary classifiers by AUC-ROC
I have a dataset with a given predictor matrix X and a response vector y (continuous values). And I also know a constant threshold to delineate y into two binary states (say 0 and 1, with y exceeding ...
0
votes
0answers
43 views
AUC and accuracy interpretation
An accuracy $80\%$ of a model that predicts binary outcomes is interpreted as:
Given a sample whose outcomes we want to predict, 80% of the prediction will be correct.
What does an AUC of $80\%$...
2
votes
1answer
60 views
Why are the trivial points included when calculating AUC?
I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue ...
0
votes
2answers
201 views
What is the formula to calculate the area under the ROC curve from a contingency table?
For example, if my table is:
...
2
votes
1answer
55 views
How to Interpret AUROC score?
My model has an AUROC value of 0.7, and I have a 75:25 class (75% negative, 25% positive) imbalance. From my understanding, AUROC is calculated by using different thresholds for considering the ...
0
votes
0answers
21 views
Insignificant t test or MW U test yet high ROC AUC and vice versa
In SPSS, when I conduct a Student's t test or Mann-Whitney U test on (lots of) variables when comparing between 2 groups, some differences are denoted significant, and others aren't.
When I conduct ...
0
votes
0answers
43 views
validation AUC systematically under 0.5 [duplicate]
I'm training a model with lightgbm (but I have the same behavior with linear regression and random forest).
I'm trying to figure out what is causing this strange training behavior.
Here my iteration ...
1
vote
2answers
81 views
Do you need to calculate sample size to evaluate a new diagnostic test?
I am writing a grant application which will be evaluating a new diagnostic test. The test will predict whether a patient with lung fibrosis will remain stable or progress. I am using an existing ...
0
votes
2answers
40 views
Performance evaluation
I'd like to test the performance of a penalized regression. I did three separate regressions for each response variable (one numerical, one binomial and one multinomial). I was checking this link, and ...
2
votes
0answers
28 views
Confidence intervals of AUC obtained by merging/pooling predictions from different test sets
I have one question regarding the CIs of the AUROC calculated merging/pooling the predictions coming from different test sets.
In one analysis, we use a sort of nested cross-validation approach, ...
1
vote
0answers
30 views
How to make really bad results from a machine learning model better by reversing predictions
I trained a classification model on some data with two classes and have really low accuracy. I have a false-positive rate of 86 % for both classes I am trying to predict. I was wondering if I could ...
0
votes
0answers
11 views
improve model roc_auc score
From the GridSearchcv on a random forest classifier, the best parameters is giving me an auc_roc score of 0.80. But when i train a new random forest model with the best parameters i am getting an ...
0
votes
0answers
15 views
which recall value to plot for same precision in PR curve?
Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:
$$[False, True, False, True, True, True, False, False],$$
which leads to the following points ...
1
vote
0answers
16 views
How to determine the optimal threshold for my classification problem using fpr ,tpr values for each classification record?
The output of my prediction using classification algorithm is in dataframe that contains TPR and FPR value for each prediction. Suppose I have 100 records for prediction then in that cade my data ...
0
votes
0answers
15 views
Directly optimizing over AUC surrogate
This is in relation to this paper
I am looking for ways to optimize Recall @ fixed Precision ($R@P$) for a machine learning problem and i didnt want to use accuracy as a proxy for $R@P$. Upon ...
0
votes
0answers
21 views
How to prove that a surrogate cost function is lower bound to original cost function?
This is very specific to a research paper that have been reading recently:
It is about constructing cost functions that are more correlated to non-decomposable (cannot be broken down to a summation ...
0
votes
0answers
31 views
Density curve in R - AUC bigger than 1
correct me if I'm wrong but I was expecting the area under the curve should be 1 for a probability density function.
Can anybody explain why it's not always the case when using the ...
0
votes
0answers
91 views
Micro- or macro-averaged AUC for highly imbalanced data?
I have a classification problem with 3 classes. With random forest classifier I'm getting the following confusion matrix:
The micro-averaged AUC is 0.76 and the macro-averaged AUC is 0.55.
On the ...
1
vote
0answers
75 views
How to interpret ROC curve with confusion matrix and F1 score?
I have implemented a random forest classifier to do a binary classification in highly imbalanced class. As the performance measurements, ROC and the f1 score was considered. However, the ROC curve ...
0
votes
0answers
40 views
How to calculate AUC for Matrix factorization
I have read paper Bayesian Personalized Ranking for implicit feedbacks (item recommendation)
Because their model is to predict that xui > xuj (xui - xuj > 0 -> xuij > 0), the paper show how to ...
1
vote
1answer
50 views
Is it possible for a model to have higher sensitivity/specificity but lower accuracy and AUC?
In the evaluation of classification models, I've found one model to have a higher accuracy and c-statistic (AUC) as compared to a second model. However, the second model has higher sensitivity, ...
1
vote
0answers
31 views
How to derive the the AUROC from the Bayes Minimum Risk (Hand 2009)?
The area-under-the-receiver-operating-characteristic-curve (AUROC) can be derived from the Bayes Minimum Risk. The derivation requires the assumption that the exact costs are unknown but follow a ...
0
votes
0answers
52 views
AUC and Accuracy baseline
I implemented the different classification algorithms like Bayesian network, Decision tree or Naive Bayes, on my data to predict the right class (binary classes). By considering confusion matrix, I ...
1
vote
0answers
34 views
Poor P-R curve for binary classifier trained on balanced data, with imbalanced test data
I have a very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing).
I am performing classification using ...
1
vote
3answers
92 views
When is an AUC score misleadingly high?
I have an algorithm which gives an AUC (area under the receiver operating curve) of 0.94.
I mean, this is amazing, but... probably too amazing, considering the difficulty of the task I am working on. ...
10
votes
2answers
394 views
Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity
I have a test set of 100 cases and two classifiers.
I generated predictions and computed ROC AUC, sensitivity and specificity for both classifiers.
Question 1: How can I compute p-value to check if ...
1
vote
1answer
146 views
Improve the precision of random forest for count data
I am trying to create a classification model that predicts whether a customer will enquire for a financial product based on some 250 independent variables. 98% of the variables are count variables and ...
0
votes
1answer
138 views
Q: Possible to optimize for area under the precision-recall curve in glmnet logistic regression?
tl;dr with the R glmnet package, is it possible to optimize for the area under the precision-recall curve, rather than the area ...
0
votes
0answers
42 views
Comparing AUCs of ROC of same diagnostic test on different samples
Supposed I've got a sample of 200 subjects, and based on these subjects, I determined the AUC of the ROC of a diagnostic test. Next, from these 200 subjects, I drew a subsample of the diagnostic test'...
1
vote
0answers
48 views
A better way to compare accuracy?
Hi I have an algorithm that takes a single sample, call it i and tries to predict what other samples in a cohort it is most closely related to. This cohort consist of N=11K from different tissues. ...
3
votes
1answer
67 views
Reconstruct ROC curve from AUC and EER
I have the values for AUC (Area under curve) and EER (Equal error rate), presented in a paper I'm reading. Is it possible to reconstruct the ROC curve from those AUC and EER values?
0
votes
0answers
98 views
What is the meaning of AUC being high when accuracy is not? [duplicate]
I'm testing several classifiers in Weka Experimenter.
Some of them have — at the same time — low accuracy (Percent_correct statistic) and high AUC.
How should the quality of such ...
1
vote
2answers
48 views
I have 2 AUCs from the same data but 2 algorithms. How I determine if one of the AUCs is greater in a statistically significant sense
Problem:
I have 200k data samples which are class imblanced (10% positive class, 90% negative class). I split the data in exactly half so my training set is 100k samples and my test set os 100k ...
2
votes
0answers
80 views
Compare and quantify relative improvement in ROC AUC scores?
What is an appropriate method for comparing relative improvement in model performance across different problems?
For example, say I have three different datasets/problems a, b, c, and two models for ...
5
votes
1answer
227 views
How is a ROCAUC=1.0 possible with imperfect accuracy? [duplicate]
I used sklearn to compute roc_auc_score for a dataset of 72 instances.
The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this ...
0
votes
1answer
45 views
What is the expected value of AUROCC for random predictions?
I was having a debate with co-workers today about the dependence of AUC on class imbalance, ie, the proportion of positive/negative instances in the response variable. It was suggested that when ...
0
votes
1answer
40 views
Post-hoc power calculation for AUC analysis, to evaluate a new diagnostic test in a cohort
I need some urgent advice for a grant application. I am evaluating a new diagnostic test and at the last minute a Professor has offered me the dataset from a completed prospective study of the disease ...
2
votes
0answers
101 views
AUC and Variable Selection
I am running into something I have not experienced and am a little confused. I have a set of about 60 predictor variables that I have manually picked from a large set. I have been running algorithms ...
