Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
-1
votes
0answers
2 views
need auc scoring with gridsearch in keras
I have unbalance dataset , i need to implement auc scoring in keras with gridsearch cv to find the best score. But in keras classifier auc and other metrics cannot be used directly , can anyone help ...
0
votes
0answers
25 views
Accuracy score or AUC extracted from Gradient Boosting Classifier of scikit-learn? [duplicate]
I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and ...
0
votes
1answer
35 views
High AUC but low R squared in a random forest classifier
I have been looking for an answer on this website and on Google but I can't seem to find a clear explanation anywhere.
The problem is the following. I built a Random Forest model (using Python's ...
1
vote
0answers
20 views
AUC depends on random splitting
I have around 1500 data samples, 10-15 features and I am training Logistic Regression for binary classification. Data is not "very clean" and I makes some imputation.
I split data into training and ...
0
votes
1answer
24 views
Stacking AUC vs. average over folds
I have a two class prediction problem where in one class I have 70% of the samples and in the other class 30% of the samples, so class imbalance. I'm conducting 10-fold cross-validation. To calcualte ...
0
votes
0answers
7 views
Testing difference of cumulative/integrated quantities of time series?
I am new to time series data analysis. I wonder how would one test the statistical significance of the difference between integrated quantities (area under curve) from different time-series curves? (...
1
vote
0answers
24 views
Comparing AUC between two subsets of test-fold
I suspect that the out-of-sample AUC of my model depends on the number of events in the out-of-sample/testing fold (I am modelling a binary variable, 0/1).
In order to test this hypothesis, I want to ...
0
votes
0answers
27 views
Aggregating ROC AUC values of several Logistic Regression Models
I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
2
votes
1answer
34 views
Area Under the Curve - Variable and Log Transformed Variable
I have a situation where I am fitting two simple logistic regression models - one model with the variable of interest included as the only predictor, and the other model with the log of the variable ...
0
votes
0answers
11 views
What is a reliable way to obtain an optimism-correct AUC with confidence limits?
I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC).
I am trying to look at a method with good performance and a plan to ...
0
votes
1answer
32 views
Is it correctly understood that ROC/AUC cannot be calculated without flexible criterion value?
I have a proprietary predictor that simply gives me a binary output. Let's say that it is detecting faulty units. In a set where 27 units are faulty and 76 units are working the predictor correctly ...
0
votes
0answers
20 views
Evaluating concordance between multiple AUC statistics measured on the same task
I am evaluating a series of features within large genomic regions.
Each feature has $n$ AUC values associated with its capacity to discriminate control from the test samples: extreme values of AUC ...
0
votes
0answers
12 views
Why is the AUC when plotted against the wilcoxon rank sum p-value, result in a plot that is not linear?
I have a binary response variable with many continuous predictor variables (about 5000). I first computed the AUC for each predictor variable, then computed the p-value associated with the AUC using ...
0
votes
1answer
24 views
Minimising CV standard deviation to increase accuracy
Does minimising the standard deviation of CV folds have any correlation with model accuracy as a theme?
I've noticed that changing the order of rows in a training data can change the AUC for each CV ...
2
votes
2answers
31 views
Why does pROC roc work with non-probability predictions?
With the pROC package, I can do this:
true <- c(1, 1, 1, 0)
predicted <- c(0.5, 0.1, 0.6, 0.1)
roc(true, predicted)
which gives as expected:
...
0
votes
1answer
38 views
High AUC and Accuracy but weird output in confusion matrix
I am working on image classification problem to determine gender given a face. The dataset is located here gender face dataset on kaggle (link to my notebook). The class distribution is as follows.
<...
0
votes
1answer
33 views
Why there is decrease in AUC values with increase in number of iteration in Held out validation?
I have a dataset with 600 rows and 4000 columns for which I am trying to do held-out cross-validation with 10 and 100 iterations.
At first, the dataset is split into 80%:20% training and “held-out” ...
0
votes
0answers
28 views
Why is XGBoost prediction proba so concentrated within specific range? (unbalanced class)
I am pretty to new to Machine Learning. I am training on some past Kaggle competitions including the Santander Customer Satisfaction Challenge (https://www.kaggle.com/c/santander-customer-satisfaction)...
1
vote
1answer
79 views
Why is cross_val_score substantially lower than .score or roc_auc_score?
I have a trained model, a GradientBoostingClassifier. My dataset is 60 thousand something rows of data that I've split into 66/33 train/test sets. Scoring the model via the ...
1
vote
0answers
25 views
Sample size for AUC based on mean and SD of raw data
I'm trying to estimate the sample size required to achieve 80% power at a 5% significance level for a superiority study comparing AUC from bioavailability data.
My challenge is the only previous ...
0
votes
0answers
44 views
Computing Standard Error of AUC over different cv folds, and assessing statistical significance
I have a dataset in a healthcare setting for which the task is to predict a binary outcome, I have done this using the Support Vector Machine algorithm in a 5-fold cross validation setup. The ...
0
votes
0answers
9 views
Which performance metric to use for stratified data? [duplicate]
I'm trying to classify a data into 3 classes (supervised), one of which is heavily underrepresented in the data set. In order to combat this imbalance, I decided to stratify the data. Now I want to ...
0
votes
1answer
172 views
How to distinguish overfitting and underfitting from the ROC AUC curve?
For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model.
But how to distinguish whether a ...
0
votes
0answers
17 views
What is a Rank in AUC in a classification method?
I have read a post on AUC for which the link is given below
What does AUC stand for and what is it?
I am putting a new post as the above is rather old.
The person gave a few points related to Rank ...
0
votes
0answers
25 views
ada model- variables overall importance
I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees.
I would like to have a ...
2
votes
1answer
44 views
K-NN optimal value of 'K'
How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score?
Do I need ...
1
vote
1answer
34 views
AUC ROC when one class consists of smaller subclasses
This question is different from Binary classification when one class consists of multiple subclasses
I have two classes that I want to distinguish by a supervised learning classifier such as a random ...
2
votes
1answer
218 views
Does AUC/ROC curve return a p-value?
When reading this article, I noticed that the legend in Figure 3 gives a p-value for each AUC (Area Under the Curve) from the ROC (Receiver Operator Characteristic) curves. It says:
The area under ...
2
votes
2answers
51 views
Appropriate way to get Cross Validated AUC
I was thinking about cross-validation and how it is the most appropriate way to do it...
Let's take the case of binary logistic regression where the goal is to calculate the AUC.
Make the partition ...
0
votes
0answers
42 views
interprete and calculate Area Under the Curve of classifier in Matlab using perfcurve?
I hope that somebody could help me with the following question. I would like to calculate the Area Under the Curve (AUC) of a classifier (Linear Discrimiant Analyses). To do this, I am using the ...
1
vote
0answers
49 views
What is the integral of the False Positive Rate over the False Positive Rate, compared to the AUC?
In machine learning the Area Under the Receiver Operating Characteristic Curve ($AUC$) can be illustrated in a plot of the True Positive Rate ($TPR$) against the False Positive Rate ($FPR$). Formally, ...
2
votes
0answers
33 views
How to calculate the standard error of ratio of two AUC
Please can anyone help me with this!
I hope my explanation is going to be simple:
This is the data I have:
I only have access to the aggregated data as below. the first column is the time points a ...
1
vote
1answer
160 views
How do I calculate AUC with leave-one-out CV
In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0
votes
0answers
26 views
High accuracy for single classes & low accuracy for multiclass classification
I was trying to do a multiclass-classification, where each sample belongs to one of the four classes. Now that I have a probability vector $(p_1^i,p_2^i,p_3^i,p_4^i)$ for each sample $i$ as my ...
2
votes
2answers
157 views
Can I trust my random forest model with low sample size and unequal class distribution?
I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution.
I am doing some explorative modeling by using 400 features to classify ...
0
votes
0answers
16 views
AUC/R^2 score strictly lower on partitioned dataset than total dataset
Suppose I've fit a logistic regression to my data and I calculate the AUC. I then partition my dataset according to one of the explanatory variables (say those rows where the variable value is greater/...
0
votes
0answers
44 views
hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R
I would like to calculate the variance of the AUC of readers (for each reader and averaged results) giving a score(1-5) to ...
2
votes
1answer
170 views
Reason for higher AUC from a test set than a training set using a random forest
I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced ...
1
vote
1answer
32 views
Why are ROC curves and AUC values not always relevant?
So I have read in posts and in literature (Frank Harrel - Regression Modeling
Strategies fx) that depending on what you do, ROC curves and AUC values are not always relevant, but often written in ...
1
vote
1answer
32 views
ROC curve: models have different FPR ranges
I've created ROC curves by calculating the TPR and FPR at various thresholds. The FPR range differs between models, so I'm wondering if AUC is still a valid way to compare the curves. A curve will ...
1
vote
1answer
105 views
How do iso accuracy line related to ROC curve
I have read many articles about ROC curve. Some specified a method to calculate the accuracy of a classifier using iso accuracy lines in a convex hull.
Just like in this articles :
http://mlwiki.org/...
2
votes
2answers
188 views
Why AUC is not a good performance metric for a classification model?
After understanding the benefits of AUC
I was stumbled to know that in some scenarios it might not be a good performance metric for evaluating a classification model. The below are the 2 scenarios:
...
2
votes
1answer
46 views
how to interpret the behavior of a ROC curve very close to one
annex the ROC curve of a logit model, which has a very high AUC, how can I interpret the behavior of this curve, I doubt the logit model?
0
votes
0answers
46 views
AUC and accuracy interpretation
An accuracy $80\%$ of a model that predicts binary outcomes is interpreted as:
Given a sample whose outcomes we want to predict, 80% of the prediction will be correct.
What does an AUC of $80\%$...
2
votes
1answer
69 views
Why are the trivial points included when calculating AUC?
I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue ...
0
votes
2answers
2k views
What is the formula to calculate the area under the ROC curve from a contingency table?
For example, if my table is:
...
2
votes
1answer
261 views
How to Interpret AUROC score?
My model has an AUROC value of 0.7, and I have a 75:25 class (75% negative, 25% positive) imbalance. From my understanding, AUROC is calculated by using different thresholds for considering the ...
0
votes
0answers
36 views
Insignificant t test or MW U test yet high ROC AUC and vice versa
In SPSS, when I conduct a Student's t test or Mann-Whitney U test on (lots of) variables when comparing between 2 groups, some differences are denoted significant, and others aren't.
When I conduct ...
1
vote
2answers
176 views
Do you need to calculate sample size to evaluate a new diagnostic test?
I am writing a grant application which will be evaluating a new diagnostic test. The test will predict whether a patient with lung fibrosis will remain stable or progress. I am using an existing ...
0
votes
2answers
42 views
Performance evaluation
I'd like to test the performance of a penalized regression. I did three separate regressions for each response variable (one numerical, one binomial and one multinomial). I was checking this link, and ...
