# Questions tagged [calibration]

Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.

316 questions
Filter by
Sorted by
Tagged with
74 views

### Can the calibration-discrimination decomposition of Brier score be viewed as the bias-variance decomposition of mean squared error?

The mean squared error has a famous decomposition into bias and variance. $$\text{MSE} = \text{bias}^2 + \text{var}$$ Brier score is also a mean squared error calculation, and Brier score has a ...
1 vote
21 views

### How to quantify the quality of a graphed calibration curve?

In his Is Medicine Mesmerized by Machine Learning? blog article, Frank Harrell shows a calibration curve (below) and states that it is quite poor. I follow the logic: the claimed probability of $0.20$...
1 vote
25 views

### Temperature scaling a bayesian neural network?

I am trying to calibrate a Bayesian neural network. I have already approximated the posterior density for its weights. In order to make predictions the Bayesian way, I am taking samples from the ...
24 views

### Calibrating a non-homogeneous Poisson process to my data [duplicate]

My question: Let's say I have some data on the cumulative number of infections per day since the start of a pandemic at $t=0$. Since clearly the infection rate changes over time, I want to calibrate a ...
1 vote
19 views

### How to diagonalise when there is less parameters to estimate than data in the Levenberg-Marquardt algorithm

I am trying to calibrate a Heston Model with 100 call options using this paper https://arxiv.org/pdf/1511.08718.pdf. In algorithm 4.1 on page 18, they define the dampening factor as: \mu_0 = \omega \...
13 views

### Comparing proxy metrics to human evaluations

I have two proxy metrics, and I'd like to see which of them correlates more strongly with human ratings. I have ~30 questions, and for each question 3 humans independently give a score on a 1-10 scale....
47 views

### XGBoost Calibration for weighted loss function

I am currently using XGBoost (in R) to perform multiclass classification. I am using merror=eval_metric and my objective is <...
16 views

### Update/ recalibrate XG Boost, Random Forest, GLM models for external validation

I have created XG Boost, Random Forest and GLM models for classification of a binary outcome and now I want to externally validate the models on a different population of over 5000 subjects. I have ...
72 views

### Do uncalibrated "probability" predictions satisfy Kolmogorov's axioms?

Let's say we have some binary variable of interest and fit a model to predict the probability of the two classes, say a logistic regression or a "classification" neural network. This model ...
1 vote
96 views

### Is perfect isotonic probability calibration realistic?

I work with a labelled tabular dataset of about 1 million observations, with the target being binary. The dataset is heavily imbalanced - about 0.5% positive class. I have trained a gradient boosting ...
1 vote
58 views

### Understanding a calibration plot for lightGBM binary classifier

I wanted to assess the performance of my lightGBM classifier using a calibration plot. If I understood correctly, a calibration plot visualizes the alignment between the predicted probabilities by the ...
1 vote
48 views

### Assessing uncertainty calibration in regression using the CDF

I have a labelled data set with $n$ data points $(x_i, y_i)$ with $x_i \in \mathbb{R}^k$ and $y_i \in \mathbb{R}$ and I trained a model $f: \mathbb{R}^k \to \mathbb{R} \times \mathbb{R}^+$ on some of ...
1 vote
26 views

### How to get standard error from constrained optimization problem in R?

Can I get standard error from a constrained optimization problem in R? I have calculated transition probabilities. Now I am trying to calibrate it. Using these transition probabilities I have ...
76 views

### Model performance with multiply imputed data

I would like to know how to do calibration plot with Hosmer-lemeshow test and AUC for ROC curve after multiple imputation in R. I build one prediction model and tried to do model performance but ...
1 vote
62 views

### Model calibration in overfitted models

Why in Shrinkage, due to an overfitted prediction model, do we tend to overestimate risk for "high risk" subjects and to underestimate risk for "low risk" subjects ? Intuitively I ...
84 views

### Optimizing a threshold value on a dependent metric using a classifier trained to optimize a threshold-independent metric

Is it a reasonable approach to train a probabilities classifier by optimizing a threshold-independent metric such as AUC, and then using the trained classifier to calibrate the decision threshold ...
73 views

### (Un)Calibrated Logistic Regression Fit

I'm fitting a logistic regression on a large dataset (n=89260, 17 predictors) with a class imbalance (1% positive class). I've tried to follow Dr. Harrell's teachings so I fit my full pre-specified ...
44 views

### Calibration plot without binning predictions

Similar to ths question I would like to know how to create a calibration curve without binning my predictions. What makes my situation different, is that I'm using icenReg for my interval-censored ...
80 views

### How does someone achieve a desired confidence / accuracy when measuring using uncalibrated instrument?

I have an instrument that measures a value. It is only possible to measure the value once i.e. the experiment can't be repeated (think recording a car's speed as it drives past). The instrument is not ...
13 views

### measure Discrimination and Calibration in case-control study

When I adopt a case-control study for risk prediction models of CC, how I can measure Discrimination and Calibration?