AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
1
vote
0answers
13 views
Is it possible for a model to have higher sensitivity/specificity but lower accuracy and AUC?
In the evaluation of classification models, I've found one model to have a higher accuracy and c-statistic (AUC) as compared to a second model. However, the second model has higher sensitivity, ...
0
votes
0answers
19 views
How to derive the the AUROC from the Bayes Minimum Risk (Hand 2009)?
The area-under-the-receiver-operating-characteristic-curve (AUROC) can be derived from the Bayes Minimum Risk. The derivation requires the assumption that the exact costs are unknown but follow a ...
-1
votes
0answers
32 views
AUC Close to 50% [closed]
I run different classification algorithms on my data on got AUC value less than 50%. For algorithm A I obtained around 60%, but algorithm B, C and D close to 50% such as 46%, 47% and 42%. So my ...
0
votes
0answers
17 views
AUC and Accuracy baseline
I implemented the different classification algorithms like Bayesian network, Decision tree or Naive Bayes, on my data to predict the right class (binary classes). By considering confusion matrix, I ...
-1
votes
0answers
17 views
Adding Feature Reduces AUC of Logistic Regression
something unexpected has come up after adding an interaction feature to a logistic regression.
In a repeated k-fold cross validation,mean accuracy and AUC are much lower than without it .The variance ...
1
vote
0answers
15 views
Poor P-R curve for binary classifier trained on balanced data, with imbalanced test data
I have a very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing).
I am performing classification using ...
1
vote
3answers
83 views
When is an AUC score misleadingly high?
I have an algorithm which gives an AUC (area under the receiver operating curve) of 0.94.
I mean, this is amazing, but... probably too amazing, considering the difficulty of the task I am working on. ...
10
votes
2answers
267 views
Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity
I have a test set of 100 cases and two classifiers.
I generated predictions and computed ROC AUC, sensitivity and specificity for both classifiers.
Question 1: How can I compute p-value to check if ...
1
vote
1answer
84 views
Improve the precision of random forest for count data
I am trying to create a classification model that predicts whether a customer will enquire for a financial product based on some 250 independent variables. 98% of the variables are count variables and ...
0
votes
1answer
52 views
Q: Possible to optimize for area under the precision-recall curve in glmnet logistic regression?
tl;dr with the R glmnet package, is it possible to optimize for the area under the precision-recall curve, rather than the area ...
0
votes
0answers
31 views
Comparing AUCs of ROC of same diagnostic test on different samples
Supposed I've got a sample of 200 subjects, and based on these subjects, I determined the AUC of the ROC of a diagnostic test. Next, from these 200 subjects, I drew a subsample of the diagnostic test'...
1
vote
0answers
42 views
A better way to compare accuracy?
Hi I have an algorithm that takes a single sample, call it i and tries to predict what other samples in a cohort it is most closely related to. This cohort consist of N=11K from different tissues. ...
3
votes
1answer
34 views
Reconstruct ROC curve from AUC and EER
I have the values for AUC (Area under curve) and EER (Equal error rate), presented in a paper I'm reading. Is it possible to reconstruct the ROC curve from those AUC and EER values?
0
votes
0answers
85 views
What is the meaning of AUC being high when accuracy is not? [duplicate]
I'm testing several classifiers in Weka Experimenter.
Some of them have — at the same time — low accuracy (Percent_correct statistic) and high AUC.
How should the quality of such ...
1
vote
2answers
47 views
I have 2 AUCs from the same data but 2 algorithms. How I determine if one of the AUCs is greater in a statistically significant sense
Problem:
I have 200k data samples which are class imblanced (10% positive class, 90% negative class). I split the data in exactly half so my training set is 100k samples and my test set os 100k ...
0
votes
0answers
14 views
How to do confounder adjustments for ROC curves (and AUC) for R/SAS [duplicate]
My example dataset contains 10 paired subjects containing true outcome, Method 1 Score, Method 2 Score and some confounder.
Outcome = c(1,0,1,1,1,1,0,0,0,1)
Method 1 = c(0, 3, 2,1,2,4,5,2,4,0)
...
2
votes
0answers
64 views
Compare and quantify relative improvement in ROC AUC scores?
What is an appropriate method for comparing relative improvement in model performance across different problems?
For example, say I have three different datasets/problems a, b, c, and two models for ...
5
votes
1answer
164 views
How is a ROCAUC=1.0 possible with imperfect accuracy?
I used sklearn to compute roc_auc_score for a dataset of 72 instances.
The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this ...
0
votes
1answer
27 views
What is the expected value of AUROCC for random predictions?
I was having a debate with co-workers today about the dependence of AUC on class imbalance, ie, the proportion of positive/negative instances in the response variable. It was suggested that when ...
0
votes
1answer
22 views
Post-hoc power calculation for AUC analysis, to evaluate a new diagnostic test in a cohort
I need some urgent advice for a grant application. I am evaluating a new diagnostic test and at the last minute a Professor has offered me the dataset from a completed prospective study of the disease ...
2
votes
0answers
57 views
AUC and Variable Selection
I am running into something I have not experienced and am a little confused. I have a set of about 60 predictor variables that I have manually picked from a large set. I have been running algorithms ...
3
votes
2answers
229 views
What to do for AUC less than 0.5?
I've trained a Random Forest model on a dataset of 60 protein predictors for healthy controls (label 0) and cancer patients (label 1).
I then tested this model on a dataset of at-risk patients ...
2
votes
0answers
26 views
Statistical test for comparing cutoff values in independent samples
A binary diagnostic test currently uses the same cutoff value (level of biomarker) for males and females when determining disease vs non-disease (D+/D-). However, I suspect that the level of biomarker ...
2
votes
3answers
132 views
AUC for random classifier in case of unbalanced dataset
If my dataset is highly unbalanced say 90% negative data point
and 10% positive data point , would using a random classifier give a AUC value of 0.5 ?
0
votes
1answer
26 views
Sample size calculation: interpreting the results for comparing AUCs
I have N number of patients each who could have 1 of 5 diseases (A, B, C, D, E). There is clinical information that may improve the accuracy of doctor diagnoses of these N patients. All diagnoses will ...
0
votes
1answer
42 views
How are AUROC scores computed with just two vectors of actual and predicted values as input? [duplicate]
In the R package ModelMetrics, the auc score as shown in the documentation takes only two inputs;
aucScore <- auc(actual=actuallabels, predicted=predictedlabels)
where the inputs are pretty self ...
2
votes
1answer
98 views
Area Under The Receiver Operating - incompatible explanations
There is one thing which confuses me about two very common explanations regarding the interpretation of the Area Under The Receiver Operating Characteristic (referred to shortly as AUC). Concretely, ...
1
vote
1answer
976 views
AUPRC vs. AUC-ROC? [duplicate]
I have come across two different terms regarding Area Under Curve (AUC):
ROC AUC: The Area Under an ROC(Receiver operating characteristic) Curve
AUPRC: The Area Under Precision-Recall Curve
Are they ...
0
votes
1answer
30 views
Can lift be lower for a model with higher AUC?
I am comparing a Deep neural net (using keras) and Xgboost on a dataset of around 3k observations with the ratio of 1's to 0's is 1:4. I am then using the models to predict on a test set and ...
0
votes
1answer
135 views
what happen if the valid set AUC higher than training AUC?
Here is the scenario . I have about 40 million instances for training, 18 million instances for testing. I use 37 million instances for training and 3 million for validation during the training. I ...
3
votes
0answers
160 views
Area Under Curve (not ROC)
I'm coming across a metrics for model evaluation which I had never seen before and I don't know how to further research (since I don't know its proper name).
I'm using someone else's code, whose goal ...
0
votes
1answer
20 views
Determine a cutpoint value of a univariate continues variable with and without modelling
I have a very simple (medical) data set with one continues independent variable X(a biomarker measurement) and y - the dependent ...
0
votes
0answers
120 views
How to compute an ROC AUC ? Why is there so many methods? [duplicate]
Disclaimer : I wasn't shure wether to post it here or on ai.staackexchange. I chose to post it here as far as there were tag on AUC and ROC
During the Kaggle Toxic Comment Classification competition ...
0
votes
0answers
36 views
Dataset imbalance problems: what is the best way to report the performance of your classifier? [duplicate]
I have a fairly simple question which is answered in many different ways all of the web, but I am having difficulty getting a single straight answer. My question relates to the best way to report a ...
0
votes
2answers
180 views
How can I plot an ROC curve? [closed]
Given this simple data:
How can I plot an ROC curve in Microsoft Excel? Step by step instructions would be very helpful. I know my TPR is ~87% and my FPR is ~13%. How do I know where to set the ...
1
vote
1answer
228 views
AUC of a cross validation model
Let's say i build a 10X cross-validation model, say with Caret.
If i want the AUC of this, is it:
The average AUC of the 10 validation samples?
Something else?
Cheers!
1
vote
0answers
74 views
Comparing area under the precision recall curve for models trained with different prevalence of the positive class
I am trying to determine the optimal prevalence of the positive class to use when training my models.
I have decided to use area under the precision recall curve (AUPRC) as my metric for determining ...
3
votes
1answer
70 views
Logistic regression: Can an additional significant predictor decrease AUC?
I recently stumbled upon a logistic regression model with four predictors $x_1$ $x_2$ $x_3$ $x_4$, having an AUC = .800 (implemented in R ...
1
vote
1answer
49 views
AUC comparison with a set of common cases
I am trying to prove there is statistical significance when I compare two classifier methods.
My proposed method only makes modifications on certain cases, the rest of them are still the same as the ...
1
vote
1answer
71 views
Distributed AUC calculation (or approximation)
I am trying to calculate the ROC AUC for a dataset where I can't fit predictions and labels in memory (10s/100s billions of samples). Is there a way to calculate the AUC in a distributed way or at ...
0
votes
1answer
108 views
leave one out cross validation and AUC less than 0.5
I have a classification problem for which I have a dataset composed by 89 istances (59 of class 0 and 30 of class 1).
Given the small dataset I perform a leave-one-out cross validation and then ...
0
votes
1answer
41 views
Logistic-regression probability score used for reordering rows for plotting ROC curve: is it the actual, positive, or negative class probability?
I want to draw my own ROC curve, such as explained in the accepted answer to this question:
Understanding ROC curve
However, it isn't clearly explained what the 'score' used represents. It is used ...
0
votes
1answer
224 views
ROC AUC and PR AUC: Are the AUC values different for each class?
When dealing with a binary classification problem, where the decision function threshold is being varied from 0 to 1 at step 0.1:
When calculating the Area Under the Curve (AUC) for a ROC Curve plot (...
6
votes
1answer
2k views
logloss vs gini/auc
I've trained two models (binary classifiers using h2o AutoML) and I want to select one to use. I have the following results:
...
1
vote
0answers
19 views
Split challenge in multiple classifiers
I am trying to solve a challenge in which I have a number of cases (in the magnitude of 100-200) and for each case I want to solve a binary classification task.
In more detail, for each case I have a ...
1
vote
1answer
2k views
What is AUC (Area Under the Curve)?
I've seen many questions posted about AUC but I'm still struggling to understand.
I see this definition for AUC everywhere "The AUC is an estimate of the probability that a classifier will rank a ...
0
votes
1answer
87 views
Does AUC Area Under Curve ROC depend on the number of original variables used in logistic regression? Does this impact reliability? [closed]
okay so I am using R(programme) to create a landslide susceptibility map - this map considers several parameters to create the map i.e. slope angle, bedrock geology. So I wanted to test the ability of ...
5
votes
2answers
149 views
How does perfect separation in logistic regression affect the AUC?
I have been working with perfect separation in logistic regression, and I have been assessing models with the AUC statistic. I was wondering what effect perfect separation has on the AUC. My own ...
0
votes
1answer
215 views
What do the thresholds on x and y axis of ROC curve represent?
There is a detailed explanation of what the AUC of an ROC curve is here. However I have searched high and low for an explanation regarding what the X and y axes of the ROC curve are. I have understood ...
0
votes
1answer
144 views
Area Under Curve for Anomaly Detection
I am performing Anomaly Detection using a CNN with a skew ratio of 1:9, and am using auc of the roc curve for evaluation of the model. The output of the CNN is a list of probabilities of positive and ...
