Questions tagged [calibration]
Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.
316
questions
4
votes
0
answers
74
views
Can the calibration-discrimination decomposition of Brier score be viewed as the bias-variance decomposition of mean squared error?
The mean squared error has a famous decomposition into bias and variance.
$$
\text{MSE} = \text{bias}^2 + \text{var}
$$
Brier score is also a mean squared error calculation, and Brier score has a ...
- 58k
1
vote
0
answers
21
views
How to quantify the quality of a graphed calibration curve?
In his Is Medicine Mesmerized by Machine Learning? blog article, Frank Harrell shows a calibration curve (below) and states that it is quite poor.
I follow the logic: the claimed probability of $0.20$...
- 58k
1
vote
0
answers
25
views
Temperature scaling a bayesian neural network?
I am trying to calibrate a Bayesian neural network. I have already approximated the posterior density for its weights. In order to make predictions the Bayesian way, I am taking samples from the ...
- 11
0
votes
0
answers
24
views
Calibrating a non-homogeneous Poisson process to my data [duplicate]
My question:
Let's say I have some data on the cumulative number of infections per day since the start of a pandemic at $t=0$. Since clearly the infection rate changes over time, I want to calibrate a ...
- 1
1
vote
1
answer
19
views
How to diagonalise when there is less parameters to estimate than data in the Levenberg-Marquardt algorithm
I am trying to calibrate a Heston Model with 100 call options using this paper https://arxiv.org/pdf/1511.08718.pdf.
In algorithm 4.1 on page 18, they define the dampening factor as: $$\mu_0 = \omega \...
0
votes
0
answers
13
views
Comparing proxy metrics to human evaluations
I have two proxy metrics, and I'd like to see which of them correlates more strongly with human ratings. I have ~30 questions, and for each question 3 humans independently give a score on a 1-10 scale....
- 101
2
votes
0
answers
47
views
XGBoost Calibration for weighted loss function
I am currently using XGBoost (in R) to perform multiclass classification. I am using merror=eval_metric and my objective is <...
- 45
0
votes
0
answers
16
views
Update/ recalibrate XG Boost, Random Forest, GLM models for external validation
I have created XG Boost, Random Forest and GLM models for classification of a binary outcome and now I want to externally validate the models on a different population of over 5000 subjects.
I have ...
- 11
0
votes
1
answer
72
views
Do uncalibrated "probability" predictions satisfy Kolmogorov's axioms?
Let's say we have some binary variable of interest and fit a model to predict the probability of the two classes, say a logistic regression or a "classification" neural network. This model ...
- 58k
1
vote
0
answers
96
views
Is perfect isotonic probability calibration realistic?
I work with a labelled tabular dataset of about 1 million observations, with the target being binary. The dataset is heavily imbalanced - about 0.5% positive class.
I have trained a gradient boosting ...
- 11
1
vote
0
answers
58
views
Understanding a calibration plot for lightGBM binary classifier
I wanted to assess the performance of my lightGBM classifier using a calibration plot. If I understood correctly, a calibration plot visualizes the alignment between the predicted probabilities by the ...
- 479
1
vote
0
answers
48
views
Assessing uncertainty calibration in regression using the CDF
I have a labelled data set with $n$ data points $(x_i, y_i)$ with $x_i \in \mathbb{R}^k$ and $y_i \in \mathbb{R}$ and I trained a model $f: \mathbb{R}^k \to \mathbb{R} \times \mathbb{R}^+$ on some of ...
- 799
1
vote
0
answers
26
views
How to get standard error from constrained optimization problem in R?
Can I get standard error from a constrained optimization problem in R? I have calculated transition probabilities. Now I am trying to calibrate it. Using these transition probabilities I have ...
0
votes
1
answer
76
views
Model performance with multiply imputed data
I would like to know how to do calibration plot with Hosmer-lemeshow test and AUC for ROC curve after multiple imputation in R. I build one prediction model and tried to do model performance but ...
1
vote
0
answers
62
views
Model calibration in overfitted models
Why in Shrinkage, due to an overfitted prediction model, do we tend to overestimate risk for "high risk" subjects and to underestimate risk for "low risk" subjects ?
Intuitively I ...
- 35
0
votes
1
answer
84
views
Optimizing a threshold value on a dependent metric using a classifier trained to optimize a threshold-independent metric
Is it a reasonable approach to train a probabilities classifier by optimizing a threshold-independent metric such as AUC, and then using the trained classifier to calibrate the decision threshold ...
- 115
0
votes
0
answers
73
views
(Un)Calibrated Logistic Regression Fit
I'm fitting a logistic regression on a large dataset (n=89260, 17 predictors) with a class imbalance (1% positive class). I've tried to follow Dr. Harrell's teachings so I fit my full pre-specified ...
- 13
0
votes
0
answers
44
views
Calibration plot without binning predictions
Similar to ths question I would like to know how to create a calibration curve without binning my predictions.
What makes my situation different, is that I'm using icenReg for my interval-censored ...
- 63
0
votes
1
answer
80
views
How does someone achieve a desired confidence / accuracy when measuring using uncalibrated instrument?
I have an instrument that measures a value.
It is only possible to measure the value once i.e. the experiment can't be repeated (think recording a car's speed as it drives past).
The instrument is not ...
- 81
0
votes
0
answers
13
views
measure Discrimination and Calibration in case-control study
When I adopt a case-control study for risk prediction models of CC, how I can measure Discrimination and Calibration?
0
votes
0
answers
51
views
At what point during model development can model calibration be applied?
I have been working on prediction models in R studio based on a rather small data set. There is a total of ~ 1200 cases with 150 to 400 positive cases depending on which of the different outcomes is ...
- 11
0
votes
0
answers
18
views
Question on Calibrated Structured Prediction (Kuleshov, 2015)
I have several questions regarding equation (2) from the paper: Calibrated Structured Prediction (Kuleshov, 2015). I believe most of my questions stem from not understanding the quantity: $T(x)=\...
1
vote
0
answers
91
views
Softmax Response vs MC Dropout for Uncertainty Estimation [closed]
Some papers I see take the uncertainty estimation of a prediction as simply its softmax/sigmoid output, whereas some papers will use techniques such as MC Dropout and calculate the variance across the ...
- 11
1
vote
0
answers
385
views
Creating and interpreting calibration plots for several models with a binary outcome
I have made several models (RF, XGB and GLM) to predict a binary outcome and they all achieved an AUC of approximately 0.8 and Brier scores 0.1-0.15.
Test set is fairly small (n= 350), cases with ...
- 11
0
votes
0
answers
214
views
AUC values of training and cross-validation are lower than AUC values of test set
I am training a Full model (logistic regression) and a few different models (LASSO, Elastic net, CART, random forest) to predict a certain clinical outcome.
I split my data into training and test sets ...
- 71
3
votes
1
answer
2k
views
Create calibration plot in R with vectors of predicted and observed values
I am currently working on a project regarding the external validation of a logistic regression model for binary classification. I would like to create a calibration plot and compute the calibration-in-...
- 31
4
votes
3
answers
521
views
Low p-value, but many trials
I am not well-versed in statistics, so please bear with me. I will try to explain things to the best of my knowledge.
I have some hardware that produces a signal, this signal has an offset and I want ...
- 43
0
votes
0
answers
10
views
Recalibrate scores across departments?
I'm looking for some assistance in the procedures to use to recalibrate subjective scores of staff performance at our university.
Say we have 10 different departments, each with 10 faculty, and each ...
0
votes
0
answers
55
views
Model for probability threshold selection, how to make differentiable loss?
I have an object detection model that outputs bounding boxes and probabilities associated with them. The probability expresses the chance that the object found is a tumor. I have noticed that the ...
- 45
2
votes
2
answers
289
views
Should sum of sample weights always be equal to the target population
I have a simple question regarding sample weighting:
From what I understand, sample weights are used to reduce or eliminate potential biases arising from the differences between the selected sample ...
- 172
3
votes
1
answer
523
views
Why do we need separate data for probability calibration?
Why do we need separate data for probability calibration?
Scikit learn documentation says:
The samples that are used to fit the calibrator should not be the same samples used to fit the classifier, as ...
- 443
1
vote
0
answers
32
views
Neural Networks Miscalibration Measure
I have read these two papers related to the neural network miscalibration problem: "On Calibration of Modern Neural Networks" and "Multivariate Confidence Calibration for Object ...
- 11
1
vote
1
answer
45
views
Does probability calibration descrease model prediction variance?
Does probability calibration decrease model prediction variance?
Example:
Let's say we have a classifier that is a mail spam detector. It outputs a score between 0-1 to quantify how likely a given ...
- 443
2
votes
0
answers
68
views
Bayesian calibration of computer simulations - Likelihood function calculation
I am starting to study Bayesian calibrations of computer models. I am not a statistician and just starting to learn so bear with me if I do not use the correct terminology.
The general approach is ...
- 121
2
votes
0
answers
279
views
Does scikit-learn support plotting calibration curve for multiclass classifier?
I have trained a multi class classifier and calibrated the classified using scikit-learn library's CalibratedClassifierCV class. To check how well the probabilities are calibrated, I tried plotting ...
- 21
0
votes
0
answers
51
views
Should I address the imbalance when using CalibratedClassifierCV?
Im using RandomForestClassifier and XGBClassifier with an imbalanced dataset, 1:2 ratio more or less, 1 being the most prevalent class.
My procedure is the following:
Use StratifiedKFold to get ...
0
votes
0
answers
35
views
Post training calibration of BNN and dataset split
since i am working with a small dataset (1048 rows) with much of the data concentrated in the region for which i have no interest, i was wondering if for Bayesian neural networks it is necessary to ...
- 325
3
votes
1
answer
2k
views
Probability Calibration for Highly Imbalanced Binary Classification
I am working on a binary classification problem on a highly imbalanced dataset (1:100) where model probabilities are important for the use case and need to be well calibrated to best represent true ...
1
vote
1
answer
52
views
Is the true conditional $P(Y \mid X)$ calibrated?
I am trying to get an intuitive understanding of the concept of calibration.
Definitions first. Consider a data distribution $P(X, Y)$ over binary labels $Y$, and a probabilistic classifier which ...
- 1,167
1
vote
0
answers
174
views
How do you do decision threshold tuning when doing k-fold cross validation?
I'm training a binary classifier for disease detection.
Because of my small amount of data (~1000 datapoints, 10% positive, 90% negative), I've realized that doing an 80-20 train-test split produces ...
- 11
1
vote
1
answer
495
views
Correct way to interpret sklearn's calibration error and generate a numeric calibration loss/score
I have been using sklearn's CalibrationDisplay and think it is pretty cool. One thing I am wondering, though, is how I could potentially take that curve and make it an interpretable score. For example,...
- 121
3
votes
1
answer
70
views
Probability Calibration, what causes simple linear models like logistic regression to be over-confident and diverge from the true class probabilities?
I've recently studied probability calibration and have investigated many examples and revisited old models of mine to find that they are all poorly calibrated. The idea of stacking also has been cast ...
- 290
0
votes
1
answer
131
views
Cross-validation curves in Cox model with time dependant covariate
I need to make cross-validation curves to check the validity of a Cox model.
So the data are divided in training and test.
The coefficients are estimated on the training and the predictor linears are ...
- 563
9
votes
2
answers
791
views
How does the Brier Score break down to (Reliability - Resolution + Uncertainty)?
The Wikipedia page states this in the decompositions section, and it is also stated in an older paper
I have never been able to understand these explanations, and I wonder what I am missing and if ...
- 812
1
vote
1
answer
68
views
Is it ok to widen a prior during an MCMC which did not converge yet?
I am calibrating parameters of a process model. The runtime of the model is high and the calibration already ran for more than two weeks with many cores on a HPC.
After almost 150k iterations I ...
- 21
1
vote
0
answers
18
views
Determine drift
I would like to determine potential drift in the data/device. Is there a general acknowledged procedure and/or method to do so?
I know how to visually "detect" a drift but I would like to ...
- 3,153
2
votes
1
answer
68
views
When developing models for effect estimation, is it advisable to perform internal validation and calibration using resampling methods? #rms
In predictive modelling, it is useful to perform resampling methods to assess performance e.g. validate() and calibrate() in the rms package in R. Is it useful / advisable to perform such methods when ...
- 499
3
votes
1
answer
231
views
How damaging to the analysis would it be to run probability validation (`rms::val.prob`) when calibration (`rms::calibrate`) is the correct action?
If I make a model that predicts probabilities (e.g., logistic regression or a neural network), I would like it to have the property that, when it predicts a probability of $p$, the event happens about ...
- 58k
1
vote
0
answers
79
views
what does it mean when the NLL becomes negative?
i was comparing the results of 3 different techniques for regression task( Deep ensembles, variational inference and concrete dropout) and i got these results
from the table looks like everything is ...
- 325
2
votes
0
answers
206
views
How to assess calibration of probability distribution for a multiclass model?
I have a multiclass classifier (boosting model), and my goal is to have a good approximation of the actual distribution to the classes given my feature values.
I.e. suppose I have features $X$, and ...
- 56