Questions tagged [calibration]

Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.

Filter by
Sorted by
Tagged with
0 votes
1 answer
29 views

Understanding Brier Loss Composition

I built several models and measured the brier loss, calibration loss, and reliability loss with the direct model and a calibrated one. Now I try to interpret the results, but I cannot understand them ...
user avatar
  • 101
0 votes
0 answers
31 views

How to do forecasting from linear regression when covariates are correlated

Consider I have in situ measurements (samples) of 4 variables: Temperature, salinity, pH, depth I know how temperature will change and want to calculate the expected change in my other variables ...
user avatar
  • 21
0 votes
0 answers
6 views

Which network to segment a rectangle on the ceiling of a room (enclosed by joists), and taking advantage of prior knowledge

I am wondering how you guys would approach this problem. Given an image from a camera pointing towards the ceiling of a room (some joists are present), I want to segment the biggest rectangular area ...
user avatar
0 votes
0 answers
11 views

Is Platt scaling applicable to a small sample size?

I am learning predictive modeling and recently came across the calibration technique called Platt scaling. I want to ask: Is this technique applicable to the small sample size such as my project (n=...
user avatar
0 votes
0 answers
11 views

Formally quantifying and assessing difference in calibration scores among different estimators?

Given a number of estimators $f_{1}, \ldots, f_{n}$ and a number of corresponding calibration scores $s_{1}, \ldots, s_{n}$ for each data group $g_{1}, \ldots, g_{m}$ with $m \ll n$. We would like to ...
user avatar
1 vote
0 answers
22 views

Machine learning tends to produce poorly-calibrated classifiers, but are the class ranks still valid?

In classification problems, "non-probabilistic" machine learning models such as boosted trees, neural networks, etc. are known to produce poorly-calibrated class scores, which aren't ...
user avatar
  • 11.5k
2 votes
1 answer
121 views

Appropriate way to get cross validated performance metrics

For cross-validation of a logistic regression classifier, it seems to me that there are a number of different approaches to calculating each performance metric: The performance metric is calculated ...
user avatar
2 votes
1 answer
70 views

Logistic Regression applied to biased dataset

I have collected a binary classification dataset in a somewhat biased way: I have thousands of unlabeled samples. A small percentage of these samples belong to the positive class. I know for a fact ...
user avatar
  • 23
2 votes
1 answer
335 views

Bootstrap optimism corrected - results interpretation

I came to know about Bootstrap validation approach for data poor settings. Currently, my problem is binary classification with 977 records and 6 features. class ratio is 77:23. Model is random forest ...
user avatar
  • 1,748
1 vote
0 answers
41 views

Why a well-calibrated model has worse brier score loss?

I already referred this post.Don't mark this as duplicate. I am working on a binary classification problem using algos like random forest, extra trees and logistic regression. dataset shape is 977, 6. ...
user avatar
  • 1,748
8 votes
1 answer
282 views

How to calibrate models if we don't have enough data?

I am working on random forest classifiation with a dataset size of 977 records and 6 features. However, my class is imbalanced and proportion is 77:23 I was reading about calibration of models (binary ...
user avatar
  • 1,748
1 vote
1 answer
38 views

How to express the agreement between experiment and theoretical observation?

Let us suppose I have a value measured from experiment and given by $$V_{\text{exp}} \pm \sigma_{V_{\text{exp}}}$$ and a theoretical value given as $$V_{\text{the}} \pm \sigma_{V_{\text{the}}}$$ Is ...
user avatar
  • 111
1 vote
1 answer
46 views

Are you always supposed to evaluate the performance of regression models?

I'm a bit confused. I am doing an analysis where there are about 70 observations of my dependent variable. I'm planning to do a multiple linear regression or multivariate logistic regression to see ...
user avatar
  • 11
0 votes
0 answers
16 views

One-vs-rest vs one-vs-one multiclass probability validation: does it matter?

Now that I have figured out how rms::val.prob works to the extent that I have written my own Python implementation, I would like to extend that idea to multiple ...
user avatar
  • 30.9k
0 votes
0 answers
10 views

Determining correct survey degrees of freedom for replicate variance estimators

Question: Recently the survey package in R adopted a clever new way to estimate the degrees of freedom for replication variance estimators: calculate the rank of ...
user avatar
  • 158
1 vote
1 answer
46 views

How to calibrate an XGBoost classifier which has been trained on a sampled dataset?

I have trained my xgboost binary classifier on a dataset which does not represent the true proportion of positive over negative observations of the population. The ...
user avatar
  • 11
0 votes
0 answers
26 views

Unbiased random walk : why is a random sample not calibrated

To simplify, consider unbiased random walks with absorbing barriers at 0 and 100. A random walk starting at X has an expected probability to hit the barrier 100 of exactly X%. However, it seems that ...
user avatar
1 vote
1 answer
187 views

Walk through rms::val.prob

The val.prob function in the rms R package has similarities to the ...
user avatar
  • 30.9k
0 votes
0 answers
17 views

Difference between ECE (expected calibration error) and ECI (estimated calibration index)

What is the difference between ECE (expected calibration error) and ECI (estimated calibration index) ? ECI: https://www.sciencedirect.com/science/article/pii/S0895435621000482
user avatar
1 vote
1 answer
108 views

Walk through `rms::calibrate` for logistic regression

The calibrate function in the rms R package allows us to compare the probability values ...
user avatar
  • 30.9k
1 vote
0 answers
22 views

Calibration of an agent based epidemic model on an HPC

I am working on HIV modeling with the EpiModel R package and we are trying to smooth our calibration process by efficiently using the Slurm clusters we have available. For reference, this is one of ...
user avatar
  • 111
0 votes
0 answers
13 views

Intuition for why a classifier cannot be well-calibrated and achieve error rate balance across groups

There are several results in the literature now, stating that a classifier cannot fulfill calibration and error rate balance at the same time if there are actual differences between groups. To pick ...
user avatar
  • 954
0 votes
0 answers
25 views

What is the name of the data set used to calibrate FPR and FNR for a model?

In spam classification domain, we often train the models on different sets than the production. Then we are faced with the problem of calibrating the models with ROC curves to find a proper cut-off ...
user avatar
6 votes
1 answer
87 views

Please clarify Bayesian calibration of the posterior mean

In the book Bayesian Data Analysis, the authors state on page 128: The concept of calibration of the posterior mean [is] the Bayesian analogue to the classical notion of bias. They define the ...
user avatar
  • 195
0 votes
0 answers
12 views

Calculate and calibrate speed of processing products

I want to define a metric -for a factory worker- with respect to how fast she/he is processing products. Assuming that she/he has to process 5 items and the time between those items is dt1, dt2, dt3, ...
user avatar
  • 159
1 vote
0 answers
9 views

Loosing performance in a model - calibration? [duplicate]

I am trying to understand what should I do when a models looses performance. Right now, when a model looses performance I am just creating a new model with new data but I’ve heard about the concept of ...
user avatar
  • 11
0 votes
2 answers
78 views

How to improve model fit of my predictive model?

I am building a predictive model and hope to improve its model fit. I have 2 predictors, BMI (continuous variable) and smoking status (binary variable), and my outcome is disease status (yes/no). Can ...
user avatar
0 votes
1 answer
12 views

What is meant by relative frequency in calibration curves

After reading docs on scikit learn on the probability calibration there's couple concerns that bug me. I don’t really understand how the curve values are calculated (y-axis) namely the frequencies. ...
user avatar
2 votes
1 answer
68 views

Can Hosmer-lemeshow chi-square statistic explain calibration?

I'm doing a logistic regression and created a calibration plot. I also conducted a Hosmer-Lemeshow test and got the corresponding chi-square. Is there any relationship between the calibration plot and ...
user avatar
1 vote
0 answers
29 views

How to choose the balance between model fit vs AUC (diagnostic accuracy)?

I would like to know how we can choose between model fit (calibration) vs AUC when building the predictive model. For example, if I have one predictor which improves the model fit but results in a ...
user avatar
0 votes
1 answer
65 views

Can I calibrate to 100% of my sample in ML regression?

I have a standard ML regression model trained on 80% of my data with 20% saved for testing. I want my model to match my full sample as best possible. Can I multiply my outputs by mean(observations ...
user avatar
1 vote
1 answer
59 views

Does Multinomial Probability Calibration Consider the Probabilities of the Non-Dominant Classes?

The gist behind Harrell's rms::calibrate function makes sense to me. While I have yet to understand the magic that lets us calculate the "true" ...
user avatar
  • 30.9k
0 votes
1 answer
55 views

Probability Calibration of Statistical Models

I am trying to better learn about the Probability Calibration of Statistical Models . For example, if a Random Forest model is trained on a binary supervised classification problem : ...
user avatar
  • 5,688
1 vote
1 answer
139 views

How to prove a loss function is classification-calibrated?

Assuming that I have a custom cross-entropy-like loss function defined as below, how can I prove that the loss function is classification-calibrated? $$ L=-\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^c w_j^{...
user avatar
2 votes
1 answer
55 views

Probability of random variable being another random variable

I am trying to understand the formalisation of the definition of calibration error of a classifier. This definition is taken from Guo et al 2017 (https://arxiv.org/abs/1706.04599): Let input $X \in \...
user avatar
1 vote
1 answer
45 views

Can I use a logistic regression as a calibration curve?

Lets use basketball as our example. The use case: There is a model that predicts the probability that the favored team will win. The probability range then is necessarily constrained between 0.5 and ...
user avatar
1 vote
1 answer
63 views

Why is binning in Expected Calibration Error done the way it is?

I see the definition of expected calibration error being $$\sum_{m=1}^{M}\frac{|B_m|}{n}|accuracy(B_m)-confidence(B_m)|$$ Where $B_m$ represents a outputs of the model that predicted class $m$ in a ...
user avatar
0 votes
1 answer
45 views

Is "probability calibration" intended to improve the performance of a statistical model?

I was watching this video over here: https://www.youtube.com/watch?v=AunotauS5yI This video brought up an interesting point that I never knew had a specific term for (i.e. probability calibration). If ...
user avatar
  • 5,688
1 vote
2 answers
541 views

How to calibrate with multiclass classification problem?

I am training a model to predict the label (target) based on loan status e.g. 0,1,2,3. So i have 4 classes. I have so far trained a model as follows: ...
user avatar
  • 299
1 vote
0 answers
182 views

Showing calibration of Bayesian credible intervals

I'd like to try and understand how one can prove that a particular strategy for assessing correctness of computational methods for Bayesian inference is sound. For a number $M$ of simulations, ...
user avatar
0 votes
0 answers
28 views

When optimizing inputs to minimize observed errors, why is it not more common to maximize one over the observed error?

Say I have a function that accepts inputs and hyper parameters to make a prediction, and I have a set of observed inputs and outputs to the true system. When I attempt to optimize the hyperparameters (...
user avatar
1 vote
0 answers
45 views

Are there methods for quantifying certainty in a non-logistic regression model? [closed]

I've been looking into certainty and how to calibrate classification models such that their predicted probability correlates well with their actual accuracy. I was wondering if such methods existed ...
user avatar
1 vote
0 answers
65 views

What is calibration of a probability model? A take using Bayes’ rule

As a discussion from last year about spam/ham email classification shows, just because a model gets perfect classification accuracy does not mean that it really knows what it's doing. In that example, ...
user avatar
  • 30.9k
2 votes
1 answer
109 views

Estimating regression optimism using the bootstrap

I am estimating optimism bias in for example risk predictions. A method for doing that is described by Frank Harrell and implemented in the R package rms. I am ...
user avatar
  • 875
0 votes
0 answers
15 views

Why there is no difference in calibration performance at its optimum operating threshold?

I trained a VGG-16 model on a highly imbalanced dataset where the positive samples (class-1) were only 20% of the negative samples (class-0) ( # positive samples: 100 and # negative samples: 500). The ...
user avatar
  • 123
0 votes
0 answers
34 views

How to interpret calibration curves in terms of class imbalance?

I am training a deep learning model toward a image classification task. The VGG-16 model is trained individually on two different training sets with varying degrees of data imbalance. Set-1 had only ...
user avatar
  • 123
0 votes
1 answer
83 views

Does calibration impact performance in a model trained with balanced dataset?

I have trained a VGG-16 model toward a binary classification task. The model was trained on equal numbers of abnormal and normal images (n=2000). Literature studies demonstrate that model calibration ...
user avatar
  • 123
1 vote
1 answer
160 views

Funky calibration plot from cph Cox regression - do I have to worry?

I have fitted a Cox regression with {rms}, N=3340, events = 2617, set time.inc at 19, the maximal time in my dataset. I used B=400 validation, followed by a B=400 ...
user avatar
  • 13
0 votes
0 answers
33 views

Calibrating non-linear regression model predictions using more recent but aggregate level actuals

We have a random forest regressor model to predict an output value ($Y_i$) associated with a particular input vector ($X_i$) (classical supervised learning setting). At the time of model training, ...
user avatar
  • 113
0 votes
0 answers
21 views

Definition of miscalibration

Question about posterior mean calibration has an excellent set of questions, and I would like to follow up about one in particular that is still bugging me. In "Prior distributions for variance ...
user avatar
  • 285

1
2 3 4 5 6