All Questions
Tagged with classification confidence-interval
50 questions
1
vote
1
answer
107
views
How to compute confidence and uncertainity of model without ground truth from softmax output?
Suppose I have 3 classes A,B,C.
Performing:
y_pred = model.predict(X) # suppose X only two sampel
Returning vector with length ...
0
votes
0
answers
25
views
How to construct class proportion confidence interval for an LLM classifier with known bias and precision and recall?
Let's say I have a dataset, $D$, with known ground truth labels. I nonetheless use a few-shot LLM classifier on this dataset to predict $k$ classes for each label.
From the LLM results, I get ...
0
votes
0
answers
13
views
Select classification model using nested cv and bootstrap auc confidence interval
My goal is to find the best 1 model out of 55 classification models.
I first ran nested cv on 55 models to see which model had better generalization. The AUC score was used as an evaluation indicator.
...
- 79
0
votes
0
answers
84
views
Confidence Interval on Probability Estimate (related to conformal prediction)
The question: I'm wondering if anyone knows of a way to obtain a confidence interval on a probability estimates obtained from a model (e.g., from a logistic regression model or a neural network) in a ...
- 123
3
votes
0
answers
49
views
Statistical Estimation of Machine Learning Metrics
Aloha Cross Validated,
In the context of binary classifiers we have a number of metrics like accuracy, precision, recall, AUROC, F1, etc. To show robustness of a model we can gain a confidence ...
- 31
0
votes
0
answers
42
views
When can the lower bound of a confidence interval be used?
I'm performing analysis on the data found in the NHANES package in R.
One of the vectors describes the estimated number of days that participants drank alcoholic beverages.
I want to classify them as ...
- 673
0
votes
1
answer
42
views
How to find the confidence level of a classification?
Say these's a random forest model used to classify 5 different classes. How do we get a confidence level of the prediction?
Say business wants to know what class will happen with high confidence
1
vote
1
answer
84
views
Improve the accuracy of one class, at the cost of other classes
Say we have a balanced image dataset of two classes of objects: green light, red light. After running deep NN classification, the model gives about 95% accuracy for both classes.
However, I need to ...
- 885
2
votes
1
answer
215
views
Bootstrapping (aleatoric and epistemic) risk score uncertainty
I am working on various risk score estimation problems. I assume individual subjects are associated with a true risk
$$r_i = f(x_i; \epsilon_i), \quad 0 \leq r_i \leq 1,$$
where $x_i$ is some ...
- 3,088
5
votes
1
answer
355
views
Risk score uncertainty quantification
I am working on various risk score estimation problems. I assume individual subjects are associated with a true risk
$$ r_i = f(x_i, \varepsilon)$$
where $x_i$ is some available information about the ...
- 3,088
1
vote
0
answers
2k
views
How to get confidence estimate of random forest model predictions?
I am working on binary classification using random forest model with dataset shape of 977, 6. Class proportion is 77:23
I built the ML model using my input data and obtained the probabilities of the ...
- 3,342
1
vote
1
answer
39
views
Model to determine when mean is unlikely to cross threshold
A doctor inserts a needle into a muscle to measure the duration of specific events. For every insertion approximately 5 data points are gathered. The doctor keeps making new insertions until he/she ...
- 11
1
vote
0
answers
123
views
Measuring the confidence of a probability prediction from a binary classifier
I've trained a binary classifier for a language identification problem. The training data is $n$ sentences from language A and $n$ sentences from language B. Such that $n$ sentences are selected ...
- 219
0
votes
1
answer
1k
views
how to caculate 95% CI for AUC? try 384 times or Hanley et al. (1982) method?
I am working on a prediction task to predict heart disease risk.
The data size is around 1500 and is splitted into train, validate and test datasets. I am use train dataset to train and use validate ...
1
vote
0
answers
136
views
Estimating Prediction Intervals for Class Probabilities in Random Forests
I have found multiple questions here (e.g. this) and great academic papers (e.g. this and this) about calculating prediction intervals for Random Forest and other techniques applied to regression ...
- 121
1
vote
1
answer
31
views
Is it correct to apply standard techniques for a confidence interval calculation to the result of a neural network?
so I have a binary classier from which I can evaluate on a test set and get a proportion (p) of which the classier has correctly gotten right. I then apply the following function to determine the 95% ...
- 43
0
votes
0
answers
326
views
Is predict_proba() reliable for SVC?
I want to implement some kind of confidence measure for my stock prediction model, which predicts the next day's trend (whether the price would rise or fall). As in, a trade order should be placed ...
- 115
4
votes
1
answer
876
views
How to compute confidence intervals for the performance estimate of nested cross validation?
Nested cross validation (NCV) is the standard procedure to estimate the performance of a classifier, after tuning its parameters and hyper-parameters. Despite being a concept quite general and widely ...
- 712
1
vote
0
answers
412
views
How to compare and get confidence intervals for paired proportions (correct detection rates) using McNemar's test
I want to compare the diagnostic efficiencies of two methods using overall correct classification rates (i.e., (sensitivity+specificity)/2, and assuming approx. ...
- 234
2
votes
3
answers
2k
views
How to compute confidence interval for Leave-one-out-cross-validation (LOOCV)
I have a very small data set of 50 samples, and I am performing LOOCV for evaluating the performance of a simple logistic regression model. I want to know the confidence interval of my evaluation, is ...
- 175
3
votes
0
answers
613
views
Confidence/Prediction Interval on binary/multiclass problems
I've recently fine-tuned a deep learning framework/model BERT for a sentiment classification task. I'd like to look at the confidence/prediction interval of the predicted sentiment scores (class 1 and ...
- 165
2
votes
3
answers
1k
views
Calculate the confidence interval of a balanced accuracy by taking the mean of the CIs of sensitivity and specificity?
Because sensitivity and specificity are typically estimated as binomial proportions (e.g. k = TP, n = TP+FN), we can use any of the methods used to estimate the confidence interval for binomial ...
- 41
0
votes
1
answer
153
views
Bootstrap classifier predictions
Let's say that I have a binary classifier and perform leave-one-out cross-validation.
I have, then, one vector of predicted $Y_{pred}$ and true $Y_{true}$ labels.
Is it correct to perform a ...
user289381
3
votes
1
answer
1k
views
Bootstrap vs Wilson score confidence interval
For estimation of the confidence interval of sensitivity and specificity, when I should use the Wilson score and when I should use bootstrapping?
- 390
1
vote
0
answers
27
views
Statistical Significance of precision WITHOUT negative labels
This is a follow-up to my earlier question here. Let there be $n$ data points (examples) to be classified into 2 classes - either positive or negative. Let's say we have a model that outputs the ...
- 161
0
votes
1
answer
121
views
Statistical significance of a classifier's precision
Suppose I have a sample of $n$ data points (examples) that have to be classified into one of two classes (positive and negative). Let's say I have a method to generate a score for each example. The ...
- 161
1
vote
0
answers
570
views
What is the notion of confidence in multi label classification?
For a single label classification, the notion of confidence is easy to understand. If the classifier has 80% confidence for 100 data points, in 80 of them the predicted label should match the actual ...
- 424
2
votes
1
answer
91
views
To improve the posterior belief receiving a time-series from a fixed data source
Let data $\mathbf{X}\in \mathbb{R}^d$ come from one of the $K$ possible sources $\mathsf{S}\in \{1,2,...,K\}$. The true $\mathsf{S}$ is unknown but it is fixed. The main task is to infer the true ...
- 526
2
votes
2
answers
1k
views
How to compute confidence interval for leave-one-out cross-validated AUC that is also repeated many times?
I have a small dataset of 100 data points and trained a random forest classifier using nested leave-one-out cross-validation. The details go like this:
In each trial out of 10:
for each leave-one-...
1
vote
0
answers
66
views
How to adjust confidence-interval based on model accuracy?
I have a binary classifier with 94% accuracy on unknown test data. I use that model to classify samples from a large population in order to infer the proportion of positives within the population. I ...
2
votes
0
answers
76
views
How can I "select the proper test sample size" to "achieve the best classification quality"?
Homework disclaimer. We were given 10k rows of sample training data.
The task is to train some well-known classifiers (as listed below),
test their performance and estimate the expected ...
- 141
3
votes
1
answer
602
views
Uncertainty and confidence in classification
I have a multi class classifier with 30-40 different classes. When i predict a class i would like to get an estimation about how certain my model is with its own prediction of that single sample . i....
- 380
2
votes
1
answer
858
views
How to combine confidence and probability scores into a single metric
I have an algorithm which outputs a confidence score and a probability score that a particular user belongs to class $C_i$, for multiple values of $i$. I want to output a single class $C_o$ as the ...
- 21
5
votes
1
answer
233
views
How to measure confidence in classifier of non-independent data?
I have some noisy high dimensional data, and each data point has a "score". The scores are roughly normally distributed. Some scores are known and some are unknown; I want to separate the unknown ...
- 63
5
votes
2
answers
3k
views
Calculating Confidence Intervals for Cross Validated Binary Classifiers
I'm experimenting with a number of models for a binary classification problem. To evaluate the performance of each model, I've used 10x repeated 10-fold cross validation to calculate the PR AUC (Area ...
- 153
2
votes
0
answers
135
views
Developing a Confidence Interval for Classification Predictions
I'm working on developing a confidence interval for classification predictions. Let's say the model I'm working on predicts whether a person defaults on a loan. I want to develop a confidence interval ...
- 331
1
vote
0
answers
43
views
Detection limit in function of measured false negative rate
Suppose that a given device returns a type II error rate of 5% when trying to detect the presence of a physical effect (in my specific case, the presence of a radioactive source, from which said ...
- 43
1
vote
0
answers
63
views
Estimating confidence interval of a test score
I have two sets of data: A and B. I want to compare the test score estimated using 10-fold cross-validation training in A with the test score when the model trained in A is tested with B. While ...
- 231
3
votes
1
answer
8k
views
Can a confidence interval be greater than 1?
I am doing a classification task and obtain an accuracy of 97.5%. Now, I calculated the confidence interval, assuming a normal distribution, at the 95% confidence level with: Accuracy +/- 1.96*...
- 227
1
vote
1
answer
2k
views
How to combine two accuracy results?
If I am doing a machine learning experiment A and my accuracy lies in the interval of 0.8 +/-0.03 and I have a machine learning experiment B with an accuracy in the interval of 0.9 +/-0.1, how can I ...
- 227
1
vote
0
answers
142
views
Is the assumption of symmetrical confidence intervals for bounded measures reasonable?
Is it legitimate to calculate confidence intervals for bounded measures like classification accuracy, AUC or correlation based on variance? Is it all right to say that the confidence interval is, for ...
- 351
3
votes
1
answer
724
views
Confidence interval for expected prediction error from cross-validation
I am using a support vector machine for binary classification on a sample of size 150 (75 of each class). I am using 5-fold stratified cross-validation to estimate the expected prediction error, i.e. ...
- 31
8
votes
3
answers
8k
views
How can I derive confidence intervals from the confusion matrix for a classifier?
I am using k-fold cross-validation to generate a confusion matrix for a classifier. I need to calculate 95% confidence intervals for the number of times each class is predicted when run against a ...
- 181
5
votes
1
answer
2k
views
Confidence intervals for the Log Loss metric for model comparison?
Quite a few Kaggle competitions have used or are using the Logarithmic Loss metric as the quality measure of a submission.
I'm wondering if there are other ways besides N-fold cross-validation to ...
- 73
0
votes
0
answers
81
views
How many samples do I need to prove that a classification algorithm is better than another?
I have two algorithms A and B, used to automatically classify each of N elements into K categories, N and K both being in the millions. Neither A or B is perfect, but it is relatively easy for a human ...
- 195
11
votes
3
answers
7k
views
Confidence interval for cross-validated classification accuracy
I'm working on a classification problem that computes a similarity metric between two input x-ray images. If the images are of the same person (label of 'right'), a higher metric will be calculated; ...
- 113
3
votes
0
answers
280
views
Role of coefficients in model selection for logistic regression
I have a model that I am using to predict mortality and it gives me an AUC of 0.799. The R code that I am using would look something like this:
...
- 1,053
12
votes
0
answers
1k
views
Computing a bootstrap confidence interval for the prediction error with the percentile and the BCa method
I have two related questions regarding the computation of a non-parametric bootstrap confidence interval for the prediction error.
Setting: I have a sample S from a data population P and a learner L, ...
- 835
3
votes
1
answer
1k
views
Confidence interval for classification error: binomial assumption vs. bootsrap resampling
I am developing a classifier using a set of N patterns, where N~1000. I am using K-fold cross-validation (with K=5) and computing the probability of classification error p (typical value is p=0.03). ...
- 31
1
vote
0
answers
2k
views
How to quantify the significance of the difference between two z-scores? [duplicate]
I have one sample and several features. I calculate a z-score for various features, and for various combinations of features. Is there a way to quantify the significance of the difference between ...
- 161