AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
0
votes
0answers
8 views
AUC and accuracy interpretation
An accuracy $80\%$ of a model that predicts binary outcomes is interpreted as:
Given a sample whose outcomes we want to predict, 80% of the prediction will be correct.
What does an AUC of $80\%$...
1
vote
1answer
29 views
Why are the trivial points included when calculating AUC?
I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue ...
0
votes
1answer
44 views
What is the formula to calculate the area under the ROC curve from a contingency table?
For example, if my table is:
...
2
votes
1answer
26 views
How to Interpret AUROC score?
My model has an AUROC value of 0.7, and I have a 75:25 class (75% negative, 25% positive) imbalance. From my understanding, AUROC is calculated by using different thresholds for considering the ...
0
votes
0answers
15 views
Insignificant t test or MW U test yet high ROC AUC and vice versa
In SPSS, when I conduct a Student's t test or Mann-Whitney U test on (lots of) variables when comparing between 2 groups, some differences are denoted significant, and others aren't.
When I conduct ...
0
votes
0answers
16 views
Compare predictive discrimination between joint model and Cox model using a AUC measure
I have two models; 1) the joint model including baseline predictors for mortality and the longitudinal biomarker as an additional predictor, and 2) a Cox model including only the baseline predictors ...
0
votes
0answers
41 views
validation AUC systematically under 0.5 [duplicate]
I'm training a model with lightgbm (but I have the same behavior with linear regression and random forest).
I'm trying to figure out what is causing this strange training behavior.
Here my iteration ...
1
vote
2answers
71 views
Do you need to calculate sample size to evaluate a new diagnostic test?
I am writing a grant application which will be evaluating a new diagnostic test. The test will predict whether a patient with lung fibrosis will remain stable or progress. I am using an existing ...
0
votes
2answers
39 views
Performance evaluation
I'd like to test the performance of a penalized regression. I did three separate regressions for each response variable (one numerical, one binomial and one multinomial). I was checking this link, and ...
2
votes
0answers
27 views
Confidence intervals of AUC obtained by merging/pooling predictions from different test sets
I have one question regarding the CIs of the AUROC calculated merging/pooling the predictions coming from different test sets.
In one analysis, we use a sort of nested cross-validation approach, ...
1
vote
0answers
30 views
How to make really bad results from a machine learning model better by reversing predictions
I trained a classification model on some data with two classes and have really low accuracy. I have a false-positive rate of 86 % for both classes I am trying to predict. I was wondering if I could ...
0
votes
0answers
8 views
improve model roc_auc score
From the GridSearchcv on a random forest classifier, the best parameters is giving me an auc_roc score of 0.80. But when i train a new random forest model with the best parameters i am getting an ...
0
votes
0answers
11 views
which recall value to plot for same precision in PR curve?
Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:
$$[False, True, False, True, True, True, False, False],$$
which leads to the following points ...
1
vote
0answers
11 views
How to determine the optimal threshold for my classification problem using fpr ,tpr values for each classification record?
The output of my prediction using classification algorithm is in dataframe that contains TPR and FPR value for each prediction. Suppose I have 100 records for prediction then in that cade my data ...
0
votes
0answers
10 views
Directly optimizing over AUC surrogate
This is in relation to this paper
I am looking for ways to optimize Recall @ fixed Precision ($R@P$) for a machine learning problem and i didnt want to use accuracy as a proxy for $R@P$. Upon ...
0
votes
0answers
12 views
How to prove that a surrogate cost function is lower bound to original cost function?
This is very specific to a research paper that have been reading recently:
It is about constructing cost functions that are more correlated to non-decomposable (cannot be broken down to a summation ...
0
votes
0answers
25 views
Density curve in R - AUC bigger than 1
correct me if I'm wrong but I was expecting the area under the curve should be 1 for a probability density function.
Can anybody explain why it's not always the case when using the ...
0
votes
0answers
24 views
Micro- or macro-averaged AUC for highly imbalanced data?
I have a classification problem with 3 classes. With random forest classifier I'm getting the following confusion matrix:
The micro-averaged AUC is 0.76 and the macro-averaged AUC is 0.55.
On the ...
1
vote
0answers
57 views
How to interpret ROC curve with confusion matrix and F1 score?
I have implemented a random forest classifier to do a binary classification in highly imbalanced class. As the performance measurements, ROC and the f1 score was considered. However, the ROC curve ...
0
votes
0answers
25 views
How to calculate AUC for Matrix factorization
I have read paper Bayesian Personalized Ranking for implicit feedbacks (item recommendation)
Because their model is to predict that xui > xuj (xui - xuj > 0 -> xuij > 0), the paper show how to ...
1
vote
1answer
43 views
Is it possible for a model to have higher sensitivity/specificity but lower accuracy and AUC?
In the evaluation of classification models, I've found one model to have a higher accuracy and c-statistic (AUC) as compared to a second model. However, the second model has higher sensitivity, ...
1
vote
0answers
27 views
How to derive the the AUROC from the Bayes Minimum Risk (Hand 2009)?
The area-under-the-receiver-operating-characteristic-curve (AUROC) can be derived from the Bayes Minimum Risk. The derivation requires the assumption that the exact costs are unknown but follow a ...
0
votes
0answers
32 views
AUC and Accuracy baseline
I implemented the different classification algorithms like Bayesian network, Decision tree or Naive Bayes, on my data to predict the right class (binary classes). By considering confusion matrix, I ...
1
vote
0answers
24 views
Poor P-R curve for binary classifier trained on balanced data, with imbalanced test data
I have a very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing).
I am performing classification using ...
1
vote
3answers
90 views
When is an AUC score misleadingly high?
I have an algorithm which gives an AUC (area under the receiver operating curve) of 0.94.
I mean, this is amazing, but... probably too amazing, considering the difficulty of the task I am working on. ...
10
votes
2answers
313 views
Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity
I have a test set of 100 cases and two classifiers.
I generated predictions and computed ROC AUC, sensitivity and specificity for both classifiers.
Question 1: How can I compute p-value to check if ...
1
vote
1answer
116 views
Improve the precision of random forest for count data
I am trying to create a classification model that predicts whether a customer will enquire for a financial product based on some 250 independent variables. 98% of the variables are count variables and ...
0
votes
1answer
84 views
Q: Possible to optimize for area under the precision-recall curve in glmnet logistic regression?
tl;dr with the R glmnet package, is it possible to optimize for the area under the precision-recall curve, rather than the area ...
0
votes
0answers
38 views
Comparing AUCs of ROC of same diagnostic test on different samples
Supposed I've got a sample of 200 subjects, and based on these subjects, I determined the AUC of the ROC of a diagnostic test. Next, from these 200 subjects, I drew a subsample of the diagnostic test'...
1
vote
0answers
44 views
A better way to compare accuracy?
Hi I have an algorithm that takes a single sample, call it i and tries to predict what other samples in a cohort it is most closely related to. This cohort consist of N=11K from different tissues. ...
3
votes
1answer
50 views
Reconstruct ROC curve from AUC and EER
I have the values for AUC (Area under curve) and EER (Equal error rate), presented in a paper I'm reading. Is it possible to reconstruct the ROC curve from those AUC and EER values?
0
votes
0answers
94 views
What is the meaning of AUC being high when accuracy is not? [duplicate]
I'm testing several classifiers in Weka Experimenter.
Some of them have — at the same time — low accuracy (Percent_correct statistic) and high AUC.
How should the quality of such ...
1
vote
2answers
48 views
I have 2 AUCs from the same data but 2 algorithms. How I determine if one of the AUCs is greater in a statistically significant sense
Problem:
I have 200k data samples which are class imblanced (10% positive class, 90% negative class). I split the data in exactly half so my training set is 100k samples and my test set os 100k ...
0
votes
0answers
18 views
How to do confounder adjustments for ROC curves (and AUC) for R/SAS [duplicate]
My example dataset contains 10 paired subjects containing true outcome, Method 1 Score, Method 2 Score and some confounder.
Outcome = c(1,0,1,1,1,1,0,0,0,1)
Method 1 = c(0, 3, 2,1,2,4,5,2,4,0)
...
2
votes
0answers
72 views
Compare and quantify relative improvement in ROC AUC scores?
What is an appropriate method for comparing relative improvement in model performance across different problems?
For example, say I have three different datasets/problems a, b, c, and two models for ...
5
votes
1answer
197 views
How is a ROCAUC=1.0 possible with imperfect accuracy? [duplicate]
I used sklearn to compute roc_auc_score for a dataset of 72 instances.
The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this ...
0
votes
1answer
36 views
What is the expected value of AUROCC for random predictions?
I was having a debate with co-workers today about the dependence of AUC on class imbalance, ie, the proportion of positive/negative instances in the response variable. It was suggested that when ...
0
votes
1answer
34 views
Post-hoc power calculation for AUC analysis, to evaluate a new diagnostic test in a cohort
I need some urgent advice for a grant application. I am evaluating a new diagnostic test and at the last minute a Professor has offered me the dataset from a completed prospective study of the disease ...
2
votes
0answers
76 views
AUC and Variable Selection
I am running into something I have not experienced and am a little confused. I have a set of about 60 predictor variables that I have manually picked from a large set. I have been running algorithms ...
3
votes
2answers
367 views
What to do for AUC less than 0.5?
I've trained a Random Forest model on a dataset of 60 protein predictors for healthy controls (label 0) and cancer patients (label 1).
I then tested this model on a dataset of at-risk patients ...
2
votes
0answers
27 views
Statistical test for comparing cutoff values in independent samples
A binary diagnostic test currently uses the same cutoff value (level of biomarker) for males and females when determining disease vs non-disease (D+/D-). However, I suspect that the level of biomarker ...
2
votes
3answers
196 views
AUC for random classifier in case of unbalanced dataset
If my dataset is highly unbalanced say 90% negative data point
and 10% positive data point , would using a random classifier give a AUC value of 0.5 ?
0
votes
1answer
27 views
Sample size calculation: interpreting the results for comparing AUCs
I have N number of patients each who could have 1 of 5 diseases (A, B, C, D, E). There is clinical information that may improve the accuracy of doctor diagnoses of these N patients. All diagnoses will ...
0
votes
1answer
79 views
How are AUROC scores computed with just two vectors of actual and predicted values as input? [duplicate]
In the R package ModelMetrics, the auc score as shown in the documentation takes only two inputs;
aucScore <- auc(actual=actuallabels, predicted=predictedlabels)
where the inputs are pretty self ...
2
votes
1answer
102 views
Area Under The Receiver Operating - incompatible explanations
There is one thing which confuses me about two very common explanations regarding the interpretation of the Area Under The Receiver Operating Characteristic (referred to shortly as AUC). Concretely, ...
1
vote
1answer
2k views
AUPRC vs. AUC-ROC? [duplicate]
I have come across two different terms regarding Area Under Curve (AUC):
ROC AUC: The Area Under an ROC(Receiver operating characteristic) Curve
AUPRC: The Area Under Precision-Recall Curve
Are they ...
0
votes
1answer
34 views
Can lift be lower for a model with higher AUC?
I am comparing a Deep neural net (using keras) and Xgboost on a dataset of around 3k observations with the ratio of 1's to 0's is 1:4. I am then using the models to predict on a test set and ...
0
votes
1answer
186 views
what happen if the valid set AUC higher than training AUC?
Here is the scenario . I have about 40 million instances for training, 18 million instances for testing. I use 37 million instances for training and 3 million for validation during the training. I ...
3
votes
0answers
171 views
Area Under Curve (not ROC)
I'm coming across a metrics for model evaluation which I had never seen before and I don't know how to further research (since I don't know its proper name).
I'm using someone else's code, whose goal ...
0
votes
1answer
22 views
Determine a cutpoint value of a univariate continues variable with and without modelling
I have a very simple (medical) data set with one continues independent variable X(a biomarker measurement) and y - the dependent ...
