Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

Filter by
Sorted by
Tagged with
3
votes
1answer
9k views

AUPRC vs. AUC-ROC? [duplicate]

I have come across two different terms regarding Area Under Curve (AUC): ROC AUC: The Area Under an ROC(Receiver operating characteristic) Curve AUPRC: The Area Under Precision-Recall Curve Are they ...
-1
votes
2answers
67 views

Can lift be lower for a model with higher AUC?

I am comparing a Deep neural net (using keras) and Xgboost on a dataset of around 3k observations with the ratio of 1's to 0's is 1:4. I am then using the models to predict on a test set and ...
0
votes
1answer
487 views

what happen if the valid set AUC higher than training AUC?

Here is the scenario . I have about 40 million instances for training, 18 million instances for testing. I use 37 million instances for training and 3 million for validation during the training. I ...
3
votes
0answers
242 views

Area Under Curve (not ROC)

I'm coming across a metrics for model evaluation which I had never seen before and I don't know how to further research (since I don't know its proper name). I'm using someone else's code, whose goal ...
0
votes
1answer
37 views

Determine a cutpoint value of a univariate continues variable with and without modelling

I have a very simple (medical) data set with one continues independent variable X(a biomarker measurement) and y - the dependent ...
0
votes
2answers
1k views

How can I plot an ROC curve? [closed]

Given this simple data: How can I plot an ROC curve in Microsoft Excel? Step by step instructions would be very helpful. I know my TPR is ~87% and my FPR is ~13%. How do I know where to set the ...
2
votes
2answers
575 views

AUC of a cross validation model

Let's say i build a 10X cross-validation model, say with Caret. If i want the AUC of this, is it: The average AUC of the 10 validation samples? Something else? Cheers!
1
vote
0answers
365 views

Comparing area under the precision recall curve for models trained with different prevalence of the positive class

I am trying to determine the optimal prevalence of the positive class to use when training my models. I have decided to use area under the precision recall curve (AUPRC) as my metric for determining ...
3
votes
1answer
132 views

Logistic regression: Can an additional significant predictor decrease AUC?

I recently stumbled upon a logistic regression model with four predictors $x_1$ $x_2$ $x_3$ $x_4$, having an AUC = .800 (implemented in R ...
1
vote
1answer
121 views

AUC comparison with a set of common cases

I am trying to prove there is statistical significance when I compare two classifier methods. My proposed method only makes modifications on certain cases, the rest of them are still the same as the ...
2
votes
1answer
318 views

Distributed AUC calculation (or approximation)

I am trying to calculate the ROC AUC for a dataset where I can't fit predictions and labels in memory (10s/100s billions of samples). Is there a way to calculate the AUC in a distributed way or at ...
0
votes
1answer
264 views

leave one out cross validation and AUC less than 0.5

I have a classification problem for which I have a dataset composed by 89 istances (59 of class 0 and 30 of class 1). Given the small dataset I perform a leave-one-out cross validation and then ...
0
votes
1answer
192 views

Logistic-regression probability score used for reordering rows for plotting ROC curve: is it the actual, positive, or negative class probability?

I want to draw my own ROC curve, such as explained in the accepted answer to this question: Understanding ROC curve However, it isn't clearly explained what the 'score' used represents. It is used ...
0
votes
1answer
980 views

ROC AUC and PR AUC: Are the AUC values different for each class?

When dealing with a binary classification problem, where the decision function threshold is being varied from 0 to 1 at step 0.1: When calculating the Area Under the Curve (AUC) for a ROC Curve plot (...
15
votes
1answer
9k views

logloss vs gini/auc

I've trained two models (binary classifiers using h2o AutoML) and I want to select one to use. I have the following results: ...
1
vote
0answers
19 views

Split challenge in multiple classifiers

I am trying to solve a challenge in which I have a number of cases (in the magnitude of 100-200) and for each case I want to solve a binary classification task. In more detail, for each case I have a ...
1
vote
1answer
6k views

What is AUC (Area Under the Curve)? [duplicate]

I've seen many questions posted about AUC but I'm still struggling to understand. I see this definition for AUC everywhere "The AUC is an estimate of the probability that a classifier will rank a ...
0
votes
1answer
150 views

Does AUC Area Under Curve ROC depend on the number of original variables used in logistic regression? Does this impact reliability? [closed]

okay so I am using R(programme) to create a landslide susceptibility map - this map considers several parameters to create the map i.e. slope angle, bedrock geology. So I wanted to test the ability of ...
5
votes
2answers
406 views

How does perfect separation in logistic regression affect the AUC?

I have been working with perfect separation in logistic regression, and I have been assessing models with the AUC statistic. I was wondering what effect perfect separation has on the AUC. My own ...
0
votes
1answer
1k views

What do the thresholds on x and y axis of ROC curve represent?

There is a detailed explanation of what the AUC of an ROC curve is here. However I have searched high and low for an explanation regarding what the X and y axes of the ROC curve are. I have understood ...
1
vote
1answer
462 views

Area Under Curve for Anomaly Detection

I am performing Anomaly Detection using a CNN with a skew ratio of 1:9, and am using auc of the roc curve for evaluation of the model. The output of the CNN is a list of probabilities of positive and ...
2
votes
3answers
825 views

AUC with incomplete ROC curve

I am doing an experiments where changing a parameter I am obtaining different number of FalsePositive, FalseNegative... and so on. I am using this parameter tuning as threshold tuning to obtain FPR ...
0
votes
2answers
184 views

Can I calculate a 95% confidence interval with 20 samples of a measure?

I am conducting a experiment that takes lots of time, and I want to report 95% CI of area under the curve (AUC) measures, is it possible to calculate this stat with only 20 samples?
1
vote
0answers
67 views

ROC Curve results remain the same

I have a dataset which has double 0' than 1'. I apply a logistic regression and a logistic regression with weights that balance the train dataset. My problem is when I create ROC curves in both cases ...
1
vote
0answers
293 views

Why an AUC of 1 is considered ideal? [duplicate]

I think I am missing something when trying to understand the meaning of ROC curve. More specifically, I don't understand why a square with AUC of 1 (100% of the area) is considered ideal. This implies ...
0
votes
1answer
156 views

Steepness of ROC vs bounded AUC to optimize fpr

As of http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossval.html The “steepness” of ROC curves is also important, since it is ideal to maximize the true positive rate while ...
0
votes
1answer
1k views

AUC in item recommendation context

I am trying to understand the AUC (Area under the ROC curve) in the context of evaluating the performance of an algorithm for doing item recommendation(e.g. BPRMF). I know how the calculation is ...
0
votes
1answer
2k views

Why does randomForest has higher test AUC than train AUC? Is this possible? [duplicate]

I am having some question with the randomForest. I use the "creatFolds" in "caret" package to partition the data into training set and test set. After building the model, I found that the test data ...
14
votes
1answer
5k views

Why use Normalized Gini Score instead of AUC as evaluation?

Kaggle's competition Porto Seguro's Safe Driver Prediction uses Normalized Gini Score as evaluation metric and this got me curious about the reasons for this choice. What are the advantages of using ...
13
votes
1answer
4k views

Comparisson of two models when the ROC curves cross each other

One common measure used to compare two or more classification models is to use the area under the ROC curve (AUC) as a way to indirectly assess their performance. In this case a model with a larger ...
5
votes
2answers
2k views

two questions; how to interpret the AUROC (area under the ROC curve)

Suppose I have fitted a Logistic regression model that predicts $P(Y=1|\boldsymbol{X})$ the presence of a disease which is encoded to $1$, and if not then $0$. The AUROC (area under the roc curve) ...
1
vote
0answers
431 views

Calculating sensitivity and specificity from survival data

I have a diagnostic test performed on 100 participants at baseline. I then follow up these participants for variable periods of time and have data regarding survival. I have used a Cox regression ...
-1
votes
1answer
116 views

How could the result of a decicion tree (maximum depth) be converted to probabilities

I would like to compute a ROC-Curve for a decision tree created with sklearn based on the CART algorithm. Logically, if I compute it with maximum depth I always get a probability of 1 or 0 (discrete ...
0
votes
0answers
761 views

AUC Scores greater than 1 with multi class classification in R? [duplicate]

I am implementing machine learning algorithm for multiclass classification problem in R programming. The problem is that when I predict the accuracy I am getting around 90% accuracy but when I ...
1
vote
1answer
444 views

What is the difference between accuracy and AUC score which one to trust?

I've 300 samples with multiclass classification problem with 3-classes. I implemented SVM in R programming. Below is the output which I am really confused. Can anyone logically explain to me what is ...
2
votes
0answers
524 views

Using bayes theorem to calculate credit risk given prior knowledge and predicted probability

How can one combine: a priori knowledge of the default proability of a certain loan type based on historical data the default probability of an individual loan as predicted by a machine learning ...
1
vote
1answer
301 views

Sensitivity/Specificity compared to Area Under Curve as measure of Screen Accuracy

In relation to the disorder I'm studying Screen A is reported as having a sensitivity of 90% and a specificity of 89%. Screen B is reported as having a AUC of .79 with no other data provided. Could ...
-1
votes
1answer
200 views

Is AUC (area under curve) a type of “empirical Bayes estimator”?

Questions: Is the AUC (area under the ROC curve) a type of "empirical Bayes estimator"? If we take the parameter space to be $\Theta = [0,1]$ and the prior $\Lambda$ to be Lebesgue measure, then the ...
3
votes
2answers
1k views

Facing unbalanced data: AUC vs. Cohen's Kappa vs. Balanced Misclassification Rate

As the question title implies, I am dealing with unbalanced data (minority class 2%) classification. As a classification tool I chose Random Forest from R package "RandomForest". So, I chose two ways ...
2
votes
2answers
62 views

Interpreting predictions of RFs based on AUCs [closed]

If I have a random forest of old independent data with an AUC of .66, a random forest of new independent data with an AUC of .75, and a random forest of old and new independent data with an AUC of .79,...
1
vote
0answers
110 views

Standard error for sum of random variables

I am conducting an analysis of pharmacokinetics data in which I have plasma concentration of drug for different dosage groups. In this way, the area under the concentration-dose curve is computed as $...
2
votes
1answer
325 views

Is there any effect of unbalanced dataset on AUROC?

I have worked on many classification problems. One of the parameters for classifier performance is AUROC/AUC which is the area under curve created by TPR and FPR values for different cutoffs of ...
0
votes
1answer
65 views

Why calculating AUC generates 'inf' value

I use the function auc in the R package pROC to calculate the ...
1
vote
0answers
98 views

Streaming auc while training

I am using the streaming auc to evaluate performance of my binary classifier. Not the regular auc because the validation is performed batch-wise and the streaming auc accumulates all aucs across all ...
2
votes
1answer
86 views

When is accuracy score preferred to AUCROC?

I have a binary and balanced dataset. Do I have to see the AUROC as the different trade-offs between the TPR and the FPR and the accuracy as a result with a threshold of 0.5? When is accuracy a ...
2
votes
1answer
581 views

ROC and accuracy results: how can AUC be one if the classifier has made mistakes?

Trying to get the accuracy and the ROC curve with R (mlr package) I get the following results: ...
5
votes
1answer
4k views

AUC score less than 0.5 for logistic regression

I've tested out various feature selection methods, such as the F-test, Mutual Information and the Extra Tree (Extra Randomised) Forest Classifier (ETC) as well as PCA (which is technically a feature ...
1
vote
0answers
926 views

ROC analysis: AUC tests in SPSS and Stata

When SPSS tests the AUC of a ROC curve against the chance area (0.5), which statistical test does it use for this? And what are its motivations to use it? This seems to be documented nowhere. Because ...
6
votes
1answer
363 views

How to get approximative confidence interval for Gini and AUC?

I found an interesting way to calculate a confidence interval for the Gini and respectively AUC coefficient for credit risk scoring. Question: Can anyone explain me, why the sum $$ AUC = \frac{1}{...
1
vote
2answers
464 views

Area Under the ROC Curve: Comparing identification performance between two values of the same variable

I'm trying to figure out ROC analysis and have the following question: Assume there is an test instrument that claims to be able to identify cats based on a series of checks (such as: does the animal ...

1 2 3
4
5
9