# Questions tagged [calibration]

Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.

259 questions
Filter by
Sorted by
Tagged with
29 views

### Understanding Brier Loss Composition

I built several models and measured the brier loss, calibration loss, and reliability loss with the direct model and a calibrated one. Now I try to interpret the results, but I cannot understand them ...
• 101
31 views

### How to do forecasting from linear regression when covariates are correlated

Consider I have in situ measurements (samples) of 4 variables: Temperature, salinity, pH, depth I know how temperature will change and want to calculate the expected change in my other variables ...
• 21
6 views

### Which network to segment a rectangle on the ceiling of a room (enclosed by joists), and taking advantage of prior knowledge

I am wondering how you guys would approach this problem. Given an image from a camera pointing towards the ceiling of a room (some joists are present), I want to segment the biggest rectangular area ...
11 views

### Is Platt scaling applicable to a small sample size?

I am learning predictive modeling and recently came across the calibration technique called Platt scaling. I want to ask: Is this technique applicable to the small sample size such as my project (n=...
• 311
11 views

### Formally quantifying and assessing difference in calibration scores among different estimators?

Given a number of estimators $f_{1}, \ldots, f_{n}$ and a number of corresponding calibration scores $s_{1}, \ldots, s_{n}$ for each data group $g_{1}, \ldots, g_{m}$ with $m \ll n$. We would like to ...
• 102
1 vote
22 views

### Machine learning tends to produce poorly-calibrated classifiers, but are the class ranks still valid?

In classification problems, "non-probabilistic" machine learning models such as boosted trees, neural networks, etc. are known to produce poorly-calibrated class scores, which aren't ...
• 11.5k
121 views

### Appropriate way to get cross validated performance metrics

For cross-validation of a logistic regression classifier, it seems to me that there are a number of different approaches to calculating each performance metric: The performance metric is calculated ...
70 views

### Logistic Regression applied to biased dataset

I have collected a binary classification dataset in a somewhat biased way: I have thousands of unlabeled samples. A small percentage of these samples belong to the positive class. I know for a fact ...
• 23
335 views

### Bootstrap optimism corrected - results interpretation

I came to know about Bootstrap validation approach for data poor settings. Currently, my problem is binary classification with 977 records and 6 features. class ratio is 77:23. Model is random forest ...
• 1,748
1 vote
41 views

### Why a well-calibrated model has worse brier score loss?

I already referred this post.Don't mark this as duplicate. I am working on a binary classification problem using algos like random forest, extra trees and logistic regression. dataset shape is 977, 6. ...
• 1,748
282 views

### How to calibrate models if we don't have enough data?

I am working on random forest classifiation with a dataset size of 977 records and 6 features. However, my class is imbalanced and proportion is 77:23 I was reading about calibration of models (binary ...
• 1,748
1 vote
38 views

### How to express the agreement between experiment and theoretical observation?

Let us suppose I have a value measured from experiment and given by $$V_{\text{exp}} \pm \sigma_{V_{\text{exp}}}$$ and a theoretical value given as $$V_{\text{the}} \pm \sigma_{V_{\text{the}}}$$ Is ...
• 111
1 vote
46 views

### Are you always supposed to evaluate the performance of regression models?

I'm a bit confused. I am doing an analysis where there are about 70 observations of my dependent variable. I'm planning to do a multiple linear regression or multivariate logistic regression to see ...
• 11
16 views

### One-vs-rest vs one-vs-one multiclass probability validation: does it matter?

Now that I have figured out how rms::val.prob works to the extent that I have written my own Python implementation, I would like to extend that idea to multiple ...
• 30.9k
10 views

### Determining correct survey degrees of freedom for replicate variance estimators

Question: Recently the survey package in R adopted a clever new way to estimate the degrees of freedom for replication variance estimators: calculate the rank of ...
• 158
1 vote
46 views

### How to calibrate an XGBoost classifier which has been trained on a sampled dataset?

I have trained my xgboost binary classifier on a dataset which does not represent the true proportion of positive over negative observations of the population. The ...
• 11
26 views

### Unbiased random walk : why is a random sample not calibrated

To simplify, consider unbiased random walks with absorbing barriers at 0 and 100. A random walk starting at X has an expected probability to hit the barrier 100 of exactly X%. However, it seems that ...
• 101
1 vote
187 views

### Walk through rms::val.prob

The val.prob function in the rms R package has similarities to the ...
• 30.9k
17 views

### Difference between ECE (expected calibration error) and ECI (estimated calibration index)

What is the difference between ECE (expected calibration error) and ECI (estimated calibration index) ? ECI: https://www.sciencedirect.com/science/article/pii/S0895435621000482
• 73
1 vote
108 views

### Walk through rms::calibrate for logistic regression

The calibrate function in the rms R package allows us to compare the probability values ...
• 30.9k
1 vote
22 views

### Calibration of an agent based epidemic model on an HPC

I am working on HIV modeling with the EpiModel R package and we are trying to smooth our calibration process by efficiently using the Slurm clusters we have available. For reference, this is one of ...
• 111
13 views

### Intuition for why a classifier cannot be well-calibrated and achieve error rate balance across groups

There are several results in the literature now, stating that a classifier cannot fulfill calibration and error rate balance at the same time if there are actual differences between groups. To pick ...
• 954
25 views

### What is the name of the data set used to calibrate FPR and FNR for a model?

In spam classification domain, we often train the models on different sets than the production. Then we are faced with the problem of calibrating the models with ROC curves to find a proper cut-off ...
• 2,092
87 views

### Please clarify Bayesian calibration of the posterior mean

In the book Bayesian Data Analysis, the authors state on page 128: The concept of calibration of the posterior mean [is] the Bayesian analogue to the classical notion of bias. They define the ...
• 195
12 views

### Calculate and calibrate speed of processing products

I want to define a metric -for a factory worker- with respect to how fast she/he is processing products. Assuming that she/he has to process 5 items and the time between those items is dt1, dt2, dt3, ...
• 159
1 vote
9 views

### Loosing performance in a model - calibration? [duplicate]

I am trying to understand what should I do when a models looses performance. Right now, when a model looses performance I am just creating a new model with new data but I’ve heard about the concept of ...
• 11
78 views

### How to improve model fit of my predictive model?

I am building a predictive model and hope to improve its model fit. I have 2 predictors, BMI (continuous variable) and smoking status (binary variable), and my outcome is disease status (yes/no). Can ...
• 311
12 views

### What is meant by relative frequency in calibration curves

After reading docs on scikit learn on the probability calibration there's couple concerns that bug me. I don’t really understand how the curve values are calculated (y-axis) namely the frequencies. ...
68 views

### Can Hosmer-lemeshow chi-square statistic explain calibration?

I'm doing a logistic regression and created a calibration plot. I also conducted a Hosmer-Lemeshow test and got the corresponding chi-square. Is there any relationship between the calibration plot and ...
• 399
1 vote
29 views

### How to choose the balance between model fit vs AUC (diagnostic accuracy)?

I would like to know how we can choose between model fit (calibration) vs AUC when building the predictive model. For example, if I have one predictor which improves the model fit but results in a ...
• 311
65 views

### Can I calibrate to 100% of my sample in ML regression?

I have a standard ML regression model trained on 80% of my data with 20% saved for testing. I want my model to match my full sample as best possible. Can I multiply my outputs by mean(observations ...
1 vote
59 views

### Does Multinomial Probability Calibration Consider the Probabilities of the Non-Dominant Classes?

The gist behind Harrell's rms::calibrate function makes sense to me. While I have yet to understand the magic that lets us calculate the "true" ...
• 30.9k
55 views

### Probability Calibration of Statistical Models

I am trying to better learn about the Probability Calibration of Statistical Models . For example, if a Random Forest model is trained on a binary supervised classification problem : ...
• 5,688
1 vote
139 views

• 640
45 views

### Is "probability calibration" intended to improve the performance of a statistical model?

I was watching this video over here: https://www.youtube.com/watch?v=AunotauS5yI This video brought up an interesting point that I never knew had a specific term for (i.e. probability calibration). If ...
• 5,688
1 vote
541 views

### How to calibrate with multiclass classification problem?

I am training a model to predict the label (target) based on loan status e.g. 0,1,2,3. So i have 4 classes. I have so far trained a model as follows: ...
• 299
1 vote
182 views

### Showing calibration of Bayesian credible intervals

I'd like to try and understand how one can prove that a particular strategy for assessing correctness of computational methods for Bayesian inference is sound. For a number $M$ of simulations, ...
28 views

### When optimizing inputs to minimize observed errors, why is it not more common to maximize one over the observed error?

Say I have a function that accepts inputs and hyper parameters to make a prediction, and I have a set of observed inputs and outputs to the true system. When I attempt to optimize the hyperparameters (...
• 237
1 vote
45 views

### Are there methods for quantifying certainty in a non-logistic regression model? [closed]

I've been looking into certainty and how to calibrate classification models such that their predicted probability correlates well with their actual accuracy. I was wondering if such methods existed ...
• 640
1 vote
65 views

### What is calibration of a probability model? A take using Bayes’ rule

As a discussion from last year about spam/ham email classification shows, just because a model gets perfect classification accuracy does not mean that it really knows what it's doing. In that example, ...
• 30.9k
109 views

### Estimating regression optimism using the bootstrap

I am estimating optimism bias in for example risk predictions. A method for doing that is described by Frank Harrell and implemented in the R package rms. I am ...
• 875
15 views

### Why there is no difference in calibration performance at its optimum operating threshold?

I trained a VGG-16 model on a highly imbalanced dataset where the positive samples (class-1) were only 20% of the negative samples (class-0) ( # positive samples: 100 and # negative samples: 500). The ...
• 123
34 views

### How to interpret calibration curves in terms of class imbalance?

I am training a deep learning model toward a image classification task. The VGG-16 model is trained individually on two different training sets with varying degrees of data imbalance. Set-1 had only ...
• 123
83 views

### Does calibration impact performance in a model trained with balanced dataset?

I have trained a VGG-16 model toward a binary classification task. The model was trained on equal numbers of abnormal and normal images (n=2000). Literature studies demonstrate that model calibration ...
• 123
1 vote
160 views

### Funky calibration plot from cph Cox regression - do I have to worry?

I have fitted a Cox regression with {rms}, N=3340, events = 2617, set time.inc at 19, the maximal time in my dataset. I used B=400 validation, followed by a B=400 ...
• 13
33 views

### Calibrating non-linear regression model predictions using more recent but aggregate level actuals

We have a random forest regressor model to predict an output value ($Y_i$) associated with a particular input vector ($X_i$) (classical supervised learning setting). At the time of model training, ...
• 113