Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
621
questions
4
votes
2
answers
351
views
How to compare two plots - an alternative for AUC?
Below I attached an example which I would like to analyze.
Let's for the example purposes assume, that these plots show the activity of two different drugs which were tested. On the Y axis we have "...
- 95
7
votes
1
answer
222
views
In cross-validation, which is the AUC population parameter I really want to estimate?
In machine learning, AUC is usually used as a performance metric of an algorithm.
As one is interested in the performance of the algorithm when applied to new cases beyond those used during the ...
5
votes
0
answers
565
views
AUC vs.Class imbalance in both training data and test data [closed]
I have three datasets as below. They are about the same prediction task.
Data 1:
Training data: 95% positive instances and 5% negative instances
Test data: 95% positive instances and 5% negative ...
- 2,095
27
votes
4
answers
52k
views
AUC and class imbalance in training/test dataset
I just start to learn the Area under the ROC curve (AUC). I am told that AUC is not reflected by data imbalance. I think it means that AUC is insensitive to imbalance in test data, rather than ...
- 2,095
5
votes
1
answer
1k
views
Why I can't optimize AUC curve with Stochastic Gradient Descent?
I am reading in different papers that it is impossible to optimize AUC in term of gradient descent, due to its non differentiability. I can't find any reference about this issue with a better ...
- 2,098
1
vote
0
answers
62
views
Wanted: Statistical test to compare array with single value [duplicate]
I am comparing ROC AUC values for different feature selection methods. Some methods are deterministic and output the same AUC score every time and some methods are non-deterministic and output ...
- 121
4
votes
0
answers
891
views
Which Bootstrap for Confidence Interval of AUC with Leave-Pair-Out-Cross-Validation?
I have to calculate the CI of the AUC (Roc) for a series of classifiers (e.g. Lasso, Random Forest, SVM) learned using the same test dataset, in order to identify the best model for this problem (...
-1
votes
1
answer
1k
views
ROC AUC score significance [duplicate]
I used the sklearn.metric roc_auc_score,it gave me a value 0.91.
What is the does this number mean?
I am interested to learn how this is calculated,could someone ...
- 101
2
votes
1
answer
895
views
imbalanced dataset
I am working on a classification problem with a highly imbalanced dataset. The ratio background to signal is about 20.
I trained an xgboost model. The ROC curve looks perfect and ROC_auc is also ...
- 211
2
votes
2
answers
1k
views
Cause of Very low AUC during 10-fold Cross-Validation
I'm using 10-fold cross-validation (CV) with $L1$-penalized Logistic Regression to estimate expected prediction error.
Briefly, each sample is part of the prediction set for exactly one fold, and I ...
- 134
1
vote
2
answers
2k
views
Calculating AUC for non-binary class
I have a dataset with a non-binary target class $c$. I want to compute the AUC of my classifier and can do this easily using the one-vs-rest approach. I train $\binom{n}{2}$ classifiers where n is the ...
- 303
2
votes
0
answers
371
views
Wilcoxon signed-rank test with AUPRC
I am a little confused about this topic.
I computed mean AUPRC values using 5-fold cross validation for my classifier. I understand that comparing the mean AUPRC values for two classifiers to ...
3
votes
1
answer
621
views
Different visualization of AUC than ROC curve
There are multiple interpretations of area under ROC curve. (e.g What does AUC stand for and what is it? ). We also know that AUC is closely related to rank correlation. Are there also different ways ...
- 7,609
1
vote
0
answers
84
views
Calculating AUC from continuous output [duplicate]
Calculating AUC for one threshold in the continuous output is simple:
AUC = (TPR - FPR + 1) / 2;
What if I want to calculate AUC for multiple thresholds in ...
- 175
0
votes
1
answer
567
views
overall AUC of two different AUC's
I have a dataset where I divide it (based on nodule longest diameter) into two subsets of cancer features.
I have got the prediction accuracy for each subset and the AUC's. I would like to combine ...
- 101
2
votes
2
answers
586
views
Logistic Regression: the logic of testing for all threshold probabilities in an ROC Plot?
In Logistic Regression, what's the logic of testing for all threshold probabilities in an ROC Plot?
I thought that in logistic regression, we're concerned with the 0.5 probability threshold, and if ...
2
votes
0
answers
131
views
How to test if there is any significance difference between the areas under under two curves [duplicate]
I am trying to test if there is any significant difference between the areas under two curves.
x-axis: length of stay (los)
y-axis: percentage
curve1: control group
curve2: intervene
Any idea how ...
- 21
6
votes
1
answer
747
views
Evaluating models by Loglogss, AUC, and Accuracy
I am evaluating three models (say, A, B, and C) by three different metrics: Log-loss, AUC, and Accuracy.
The results show that
Log-loss: C>A>B (B has the best performance in terms of Log-loss)
...
- 2,095
2
votes
3
answers
2k
views
The meaning of slope on a ROC curve
I think I should rephrase my question after reading a few replies. The original question is kept intact at the bottom. So maybe I should ask the question this way: if you use a ROC curve to find the ...
- 71
1
vote
2
answers
65
views
Can the AUCs of ROCs for different sexes distinguish between sexual attitudes?
I'm not a statistician but I've been reading all about ROC and AUC today and I"m getting closer but still don't quite get it.
If I have a model and I want to know if it's penalizing men or women ...
- 11
1
vote
0
answers
728
views
Is sensitivity, specificity and g-mean considered as "point-wise" metrics
I have two questions:
just read this answer and I don't think I totally understand this term ...
does sensitivity and specificity and other measures derived from these two such as the geometric ...
- 161
2
votes
2
answers
6k
views
Why does sklearn and tensorflow give different ROC AUC scores?
My tensorflow ML algorithm gives me an ROC AUC of 0.81 using the contrib.metrics.streaming_auc() function, whereas using the same logits and labels in sklearn's function gives me a score of 0.58. How ...
- 151
0
votes
2
answers
843
views
Comparison of AUC below 0.5?
I have two sets (small) of features that I need to compare against a class, and decide which set is better at classifying. The features are generated using ranking methods, and only the top 2% of the ...
- 1
1
vote
1
answer
1k
views
ROC curve: how come I have an awful AUC (0.57) with a significant p value (<.001)?
I'm trying to identify a cut-off in a sample of values (areas): I'd like to state "if you encounter an area greater than... you can suspect a pathologic condition" or something like this. It's a case-...
- 13
1
vote
2
answers
2k
views
Why the predicted probalility in logistic regression is small, spreading within (0.0, 0.6), rather than(0.0, 1.0)
Just realized that the predicted probability of default in logistic regression is small and spread within a narrow range. My first guess is that it could be related to the not-strong-features selected ...
- 11
14
votes
4
answers
9k
views
Is AUC the probability of correctly classifying a randomly selected instance from each class?
I read this caption in a paper and have never seen AUC described in this way anywhere else. Is this true? Is there a proof or simple way to see this?
Fig. 2 shows the prediction accuracy of ...
- 1,955
1
vote
0
answers
172
views
Should Precision-Recall Area Under the Curve be similar when running binary classification and isolating a OnevsRest class AUC?
I am currently using sklearn OnevsRest classifier on a dataset. The observations of my datasets are describing actual mammals and the columns of the datasets are ...
- 11
1
vote
2
answers
3k
views
ROC accuracy measure error when no positive values
I have an unbalanced dataset that I am working to either up or down sample to balance out. But in the meantime I've run into an interesting error when calculating the AUC in R using ...
- 1,008
22
votes
2
answers
16k
views
optimizing auc vs logloss in binary classification problems
I am performing a binary classification task where the outcome probability is fair low (around 3 per cent). I am trying to decide whether to optimize by AUC or log-loss. As much as I have understood, ...
- 3,692
0
votes
1
answer
16k
views
How to improve auc of a classifier? [closed]
I have a data set with binary imbalanced class problem. Only 12% of the records belong to positive class. The auc of the original dataset for many classifiers were around 0.6 or less. So I applied ...
- 113
5
votes
2
answers
3k
views
impact to AUC if swap positive and negative during model training
If I swap positive class and negative class, then train a model again (I tried decision tree, adaboost, svm from scikit-learn built-in package) for a two class classification problem. Sometimes, I can ...
- 237
0
votes
2
answers
474
views
Problem in interpreting AUC confidence interval
I calculated AUC using SPSS and the output was:
0.623, std.Error=0.056. p=0.055. lower Bound was 0.513 and upper was 0.743
How could it be that the p value is not significant and the lower Bound of C....
1
vote
0
answers
76
views
How to compare the performance of different feature subsets with the same classifier? [duplicate]
I have a small dataset (55 samples) described by 20 features.
I performed a SVM (RBF) approach with cross-validation on 70% of the dataset (training part) and I recorded the AUC (average) for 150 ...
- 69
3
votes
0
answers
74
views
Efficient AUC Calcuation
I have to calculate the average AUC over several million AUCs and was researching on whether there is a more efficient way to do it that doesn't involve sorting. Even an approximation of the ...
- 504
6
votes
3
answers
966
views
Correctly expressing improvement in AUC?
If a set of features, A, results in an AUC of 0.5, then an improved set of features, B, were used resulting in an AUC of 0.75, how do I express this improvement in words:
50% improvement (0.75-0.5)?...
- 61
1
vote
1
answer
2k
views
Choosing the correct AUC value with RocR package
I have somewhat of an odd question.
I'm currently working on a dataset and after training my model, I'm trying to get the correct AUC and I'm using the RocR package to get that value.
Im using ...
- 123
1
vote
1
answer
344
views
Kappa and downsampling, selection of data set
I have a unbalanced data set and use Cohen's kappa and AUC as performance measure.
Without down sampling the Kappa value is around 0.85, with random down sampling it is 0.95. and with a house-made ...
- 343
3
votes
2
answers
1k
views
What is the effect of training a model on an imbalanced dataset & using it on a balanced dataset?
When evaluating a model, for example a binary classifier, should the train and test set have 50% + and 50% - label distribution or could the distribution be random?
If the distribution is biased in ...
- 71
1
vote
1
answer
600
views
What metric to use as the cross validation error in the training set for a binary classification problem?
When I am running cross validation on the training set for a binary classification problem, what metric should I use if I am only interested in obtaining the largest AUC (area under receiver operating ...
- 449
18
votes
2
answers
11k
views
Accuracy vs. area under the ROC curve
I constructed an ROC curve for a diagnostic system. The area under the curve was then non-parametrically estimated to be AUC = 0.89. When I tried to calculate the accuracy at the optimum threshold ...
- 593
3
votes
2
answers
4k
views
AUROC equal to 1.0 means overfitting?
Evaluating the classifier I implemented for university, I am observing an AUROC (Area under curve of the ROC) of 1.0 (which means a TP rate of 1 and a FP rate of 0.0)
The dataset used for training ...
- 151
9
votes
3
answers
8k
views
Is overfitted model with higher AUC on test sample better than not overfitted one
i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set.
The other participant has created a model that performs 96% AUC ...
- 177
1
vote
1
answer
366
views
AUC seems too high, confusion matrix seems only slightly better than random
My confusion matrix looks as follows:
> table(actual, predicted_all)
predicted_all
actual 0 1
0 1728 5261
1 2088 168
While the AUC ...
- 171
2
votes
0
answers
652
views
Parametric versus nonparametric AUC from ROC curve
SPSS offers two ways to estimate Area Under the Curve (AUC) and its standard error, that is nonparametricly using trapezoidal rule or parametrically using binegative exponential distribution. I have ...
- 855
1
vote
1
answer
1k
views
Does AUC for multiple logistic regression make sense if prediction is not the goal?
Does it makes sense to calculate the AUC if I do not want to use my multiple logistic regression model for predictions? I only want to calculate some odds ratios and test if the variables in my model ...
- 21
6
votes
1
answer
3k
views
How to compute the AUROC for a single categorical variable
I am building new features for a binary classifier. The new features fall into two categories: categorical and ordinal. An example of the first feature would be the colours ...
- 4,392
8
votes
1
answer
2k
views
outlier detection: area under precision recall curve
I would like to compare outlier detection algorithms. I am not sure if area under roc or under precision recall curve is the measure to use.
A quick test in matlab gives me strange results. I try to ...
- 213
6
votes
2
answers
3k
views
The distribution of the AUC
I am wondering how the confidence interval for the Area under the Curve statistic (ROC curves) is derived. I have heard that the AUC can be assumed to be normally distributed, but I am looking for a ...
- 1,481
2
votes
2
answers
2k
views
How much is ROC biased towards the minority class?
It's known that ROC is overly optimistic in case of imbalanced data sets. How big can this bias be? For example if I read paper where they report 0.75 ROC on a dataset with 5 percent of samples being ...
- 7,609
3
votes
2
answers
3k
views
Results from rfe function (caret) to compute average metrics - R
I am computing a SVM-RFE model with the rfe function of the caret package, but I am a bit confused about the results. My code is:...
- 55