Questions tagged [calibration]
Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.
259
questions
0
votes
1
answer
29
views
Understanding Brier Loss Composition
I built several models and measured the brier loss, calibration loss, and reliability loss with the direct model and a calibrated one. Now I try to interpret the results, but I cannot understand them ...
- 101
0
votes
0
answers
31
views
How to do forecasting from linear regression when covariates are correlated
Consider I have in situ measurements (samples) of 4 variables: Temperature, salinity, pH, depth
I know how temperature will change and want to calculate the expected change in my other variables ...
- 21
0
votes
0
answers
6
views
Which network to segment a rectangle on the ceiling of a room (enclosed by joists), and taking advantage of prior knowledge
I am wondering how you guys would approach this problem.
Given an image from a camera pointing towards the ceiling of a room (some joists are present), I want to segment the biggest rectangular area ...
0
votes
0
answers
11
views
Is Platt scaling applicable to a small sample size?
I am learning predictive modeling and recently came across the calibration technique called Platt scaling. I want to ask:
Is this technique applicable to the small sample size such as my project (n=...
- 311
0
votes
0
answers
11
views
Formally quantifying and assessing difference in calibration scores among different estimators?
Given a number of estimators $f_{1}, \ldots, f_{n}$ and a number of corresponding calibration scores $s_{1}, \ldots, s_{n}$ for each data group $g_{1}, \ldots, g_{m}$ with $m \ll n$. We would like to ...
- 102
1
vote
0
answers
22
views
Machine learning tends to produce poorly-calibrated classifiers, but are the class ranks still valid?
In classification problems, "non-probabilistic" machine learning models such as boosted trees, neural networks, etc. are known to produce poorly-calibrated class scores, which aren't ...
- 11.5k
2
votes
1
answer
121
views
Appropriate way to get cross validated performance metrics
For cross-validation of a logistic regression classifier, it seems to me that there are a number of different approaches to calculating each performance metric:
The performance metric is calculated ...
- 144
2
votes
1
answer
70
views
Logistic Regression applied to biased dataset
I have collected a binary classification dataset in a somewhat biased way:
I have thousands of unlabeled samples.
A small percentage of these samples belong to the positive class.
I know for a fact ...
- 23
2
votes
1
answer
335
views
Bootstrap optimism corrected - results interpretation
I came to know about Bootstrap validation approach for data poor settings. Currently, my problem is binary classification with 977 records and 6 features. class ratio is 77:23. Model is random forest
...
- 1,748
1
vote
0
answers
41
views
Why a well-calibrated model has worse brier score loss?
I already referred this post.Don't mark this as duplicate.
I am working on a binary classification problem using algos like random forest, extra trees and logistic regression. dataset shape is 977, 6. ...
- 1,748
8
votes
1
answer
282
views
How to calibrate models if we don't have enough data?
I am working on random forest classifiation with a dataset size of 977 records and 6 features. However, my class is imbalanced and proportion is 77:23
I was reading about calibration of models (binary ...
- 1,748
1
vote
1
answer
38
views
How to express the agreement between experiment and theoretical observation?
Let us suppose I have a value measured from experiment and given by
$$V_{\text{exp}} \pm \sigma_{V_{\text{exp}}}$$ and a theoretical value given as
$$V_{\text{the}} \pm \sigma_{V_{\text{the}}}$$
Is ...
- 111
1
vote
1
answer
46
views
Are you always supposed to evaluate the performance of regression models?
I'm a bit confused. I am doing an analysis where there are about 70 observations of my dependent variable. I'm planning to do a multiple linear regression or multivariate logistic regression to see ...
- 11
0
votes
0
answers
16
views
One-vs-rest vs one-vs-one multiclass probability validation: does it matter?
Now that I have figured out how rms::val.prob works to the extent that I have written my own Python implementation, I would like to extend that idea to multiple ...
- 30.9k
0
votes
0
answers
10
views
Determining correct survey degrees of freedom for replicate variance estimators
Question:
Recently the survey package in R adopted a clever new way to estimate the degrees of freedom for replication variance estimators: calculate the rank of ...
- 158
1
vote
1
answer
46
views
How to calibrate an XGBoost classifier which has been trained on a sampled dataset?
I have trained my xgboost binary classifier on a dataset which does not represent the true proportion of positive over negative observations of the population. The ...
- 11
0
votes
0
answers
26
views
Unbiased random walk : why is a random sample not calibrated
To simplify, consider unbiased random walks with absorbing barriers at 0 and 100. A random walk starting at X has an expected probability to hit the barrier 100 of exactly X%.
However, it seems that ...
- 101
1
vote
1
answer
187
views
Walk through rms::val.prob
The val.prob function in the rms R package has similarities to the ...
- 30.9k
0
votes
0
answers
17
views
Difference between ECE (expected calibration error) and ECI (estimated calibration index)
What is the difference between ECE (expected calibration error) and ECI (estimated calibration index) ?
ECI: https://www.sciencedirect.com/science/article/pii/S0895435621000482
- 73
1
vote
1
answer
108
views
Walk through `rms::calibrate` for logistic regression
The calibrate function in the rms R package allows us to compare the probability values ...
- 30.9k
1
vote
0
answers
22
views
Calibration of an agent based epidemic model on an HPC
I am working on HIV modeling with the
EpiModel R package and we are trying to smooth our calibration process by efficiently using the
Slurm clusters we have available. For reference, this is one of ...
- 111
0
votes
0
answers
13
views
Intuition for why a classifier cannot be well-calibrated and achieve error rate balance across groups
There are several results in the literature now, stating that a classifier cannot fulfill calibration and error rate balance at the same time if there are actual differences between groups. To pick ...
- 954
0
votes
0
answers
25
views
What is the name of the data set used to calibrate FPR and FNR for a model?
In spam classification domain, we often train the models on different sets than the production. Then we are faced with the problem of calibrating the models with ROC curves to find a proper cut-off ...
- 2,092
6
votes
1
answer
87
views
Please clarify Bayesian calibration of the posterior mean
In the book Bayesian Data Analysis, the authors state on page 128:
The concept of calibration of the posterior mean [is] the Bayesian analogue to the classical notion of bias.
They define the ...
- 195
0
votes
0
answers
12
views
Calculate and calibrate speed of processing products
I want to define a metric -for a factory worker- with respect to how fast she/he is processing products.
Assuming that she/he has to process 5 items and the time between those items is dt1, dt2, dt3, ...
- 159
1
vote
0
answers
9
views
Loosing performance in a model - calibration? [duplicate]
I am trying to understand what should I do when a models looses performance. Right now, when a model looses performance I am just creating a new model with new data but I’ve heard about the concept of ...
- 11
0
votes
2
answers
78
views
How to improve model fit of my predictive model?
I am building a predictive model and hope to improve its model fit. I have 2 predictors, BMI (continuous variable) and smoking status (binary variable), and my outcome is disease status (yes/no).
Can ...
- 311
0
votes
1
answer
12
views
What is meant by relative frequency in calibration curves
After reading docs on scikit learn on the probability calibration there's couple concerns that bug me. I don’t really understand how the curve values are calculated (y-axis) namely the frequencies. ...
2
votes
1
answer
68
views
Can Hosmer-lemeshow chi-square statistic explain calibration?
I'm doing a logistic regression and created a calibration plot. I also conducted a Hosmer-Lemeshow test and got the corresponding chi-square. Is there any relationship between the calibration plot and ...
- 399
1
vote
0
answers
29
views
How to choose the balance between model fit vs AUC (diagnostic accuracy)?
I would like to know how we can choose between model fit (calibration) vs AUC when building the predictive model. For example, if I have one predictor which improves the model fit but results in a ...
- 311
0
votes
1
answer
65
views
Can I calibrate to 100% of my sample in ML regression?
I have a standard ML regression model trained on 80% of my data with 20% saved for testing.
I want my model to match my full sample as best possible.
Can I multiply my outputs by mean(observations ...
- 3
1
vote
1
answer
59
views
Does Multinomial Probability Calibration Consider the Probabilities of the Non-Dominant Classes?
The gist behind Harrell's rms::calibrate function makes sense to me. While I have yet to understand the magic that lets us calculate the "true" ...
- 30.9k
0
votes
1
answer
55
views
Probability Calibration of Statistical Models
I am trying to better learn about the Probability Calibration of Statistical Models . For example, if a Random Forest model is trained on a binary supervised classification problem :
...
- 5,688
1
vote
1
answer
139
views
How to prove a loss function is classification-calibrated?
Assuming that I have a custom cross-entropy-like loss function defined as below, how can I prove that the loss function is classification-calibrated?
$$
L=-\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^c w_j^{...
- 65
2
votes
1
answer
55
views
Probability of random variable being another random variable
I am trying to understand the formalisation of the definition of calibration error of a classifier.
This definition is taken from Guo et al 2017 (https://arxiv.org/abs/1706.04599):
Let input $X \in \...
- 761
1
vote
1
answer
45
views
Can I use a logistic regression as a calibration curve?
Lets use basketball as our example.
The use case:
There is a model that predicts the probability that the favored team will win. The probability range then is necessarily constrained between 0.5 and ...
- 223
1
vote
1
answer
63
views
Why is binning in Expected Calibration Error done the way it is?
I see the definition of expected calibration error being
$$\sum_{m=1}^{M}\frac{|B_m|}{n}|accuracy(B_m)-confidence(B_m)|$$
Where $B_m$ represents a outputs of the model that predicted class $m$ in a ...
- 640
0
votes
1
answer
45
views
Is "probability calibration" intended to improve the performance of a statistical model?
I was watching this video over here: https://www.youtube.com/watch?v=AunotauS5yI
This video brought up an interesting point that I never knew had a specific term for (i.e. probability calibration).
If ...
- 5,688
1
vote
2
answers
541
views
How to calibrate with multiclass classification problem?
I am training a model to predict the label (target) based on loan status e.g. 0,1,2,3. So i have 4 classes. I have so far trained a model as follows:
...
- 299
1
vote
0
answers
182
views
Showing calibration of Bayesian credible intervals
I'd like to try and understand how one can prove that a particular strategy for assessing correctness of computational methods for Bayesian inference is sound.
For a number $M$ of simulations, ...
0
votes
0
answers
28
views
When optimizing inputs to minimize observed errors, why is it not more common to maximize one over the observed error?
Say I have a function that accepts inputs and hyper parameters to make a prediction, and I have a set of observed inputs and outputs to the true system. When I attempt to optimize the hyperparameters (...
- 237
1
vote
0
answers
45
views
Are there methods for quantifying certainty in a non-logistic regression model? [closed]
I've been looking into certainty and how to calibrate classification models such that their predicted probability correlates well with their actual accuracy. I was wondering if such methods existed ...
- 640
1
vote
0
answers
65
views
What is calibration of a probability model? A take using Bayes’ rule
As a discussion from last year about spam/ham email classification shows, just because a model gets perfect classification accuracy does not mean that it really knows what it's doing. In that example, ...
- 30.9k
2
votes
1
answer
109
views
Estimating regression optimism using the bootstrap
I am estimating optimism bias in for example risk predictions. A method for doing that is described by Frank Harrell and implemented in the R package rms. I am ...
- 875
0
votes
0
answers
15
views
Why there is no difference in calibration performance at its optimum operating threshold?
I trained a VGG-16 model on a highly imbalanced dataset where the positive samples (class-1) were only 20% of the negative samples (class-0) ( # positive samples: 100 and # negative samples: 500). The ...
- 123
0
votes
0
answers
34
views
How to interpret calibration curves in terms of class imbalance?
I am training a deep learning model toward a image classification task. The VGG-16 model is trained individually on two different training sets with varying degrees of data imbalance. Set-1 had only ...
- 123
0
votes
1
answer
83
views
Does calibration impact performance in a model trained with balanced dataset?
I have trained a VGG-16 model toward a binary classification task. The model was trained on equal numbers of abnormal and normal images (n=2000). Literature studies demonstrate that model calibration ...
- 123
1
vote
1
answer
160
views
Funky calibration plot from cph Cox regression - do I have to worry?
I have fitted a Cox regression with {rms}, N=3340, events = 2617, set time.inc at 19, the maximal time in my dataset. I used B=400 validation, followed by a B=400 ...
- 13
0
votes
0
answers
33
views
Calibrating non-linear regression model predictions using more recent but aggregate level actuals
We have a random forest regressor model to predict an output value ($Y_i$) associated with a particular input vector ($X_i$) (classical supervised learning setting).
At the time of model training, ...
- 113
0
votes
0
answers
21
views
Definition of miscalibration
Question about posterior mean calibration has an excellent set of questions, and I would like to follow up about one in particular that is still bugging me. In "Prior distributions for variance ...
- 285