Questions tagged [calibration]

Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.

Filter by
Sorted by
Tagged with
4 votes
0 answers
74 views

Can the calibration-discrimination decomposition of Brier score be viewed as the bias-variance decomposition of mean squared error?

The mean squared error has a famous decomposition into bias and variance. $$ \text{MSE} = \text{bias}^2 + \text{var} $$ Brier score is also a mean squared error calculation, and Brier score has a ...
Dave's user avatar
  • 58k
1 vote
0 answers
21 views

How to quantify the quality of a graphed calibration curve?

In his Is Medicine Mesmerized by Machine Learning? blog article, Frank Harrell shows a calibration curve (below) and states that it is quite poor. I follow the logic: the claimed probability of $0.20$...
Dave's user avatar
  • 58k
1 vote
0 answers
25 views

Temperature scaling a bayesian neural network?

I am trying to calibrate a Bayesian neural network. I have already approximated the posterior density for its weights. In order to make predictions the Bayesian way, I am taking samples from the ...
Randomdude's user avatar
0 votes
0 answers
24 views

Calibrating a non-homogeneous Poisson process to my data [duplicate]

My question: Let's say I have some data on the cumulative number of infections per day since the start of a pandemic at $t=0$. Since clearly the infection rate changes over time, I want to calibrate a ...
Michaël's user avatar
1 vote
1 answer
19 views

How to diagonalise when there is less parameters to estimate than data in the Levenberg-Marquardt algorithm

I am trying to calibrate a Heston Model with 100 call options using this paper https://arxiv.org/pdf/1511.08718.pdf. In algorithm 4.1 on page 18, they define the dampening factor as: $$\mu_0 = \omega \...
THAT'S MY QUANT MY QUANTITATIV's user avatar
0 votes
0 answers
13 views

Comparing proxy metrics to human evaluations

I have two proxy metrics, and I'd like to see which of them correlates more strongly with human ratings. I have ~30 questions, and for each question 3 humans independently give a score on a 1-10 scale....
augray's user avatar
  • 101
2 votes
0 answers
47 views

XGBoost Calibration for weighted loss function

I am currently using XGBoost (in R) to perform multiclass classification. I am using merror=eval_metric and my objective is <...
HeyCool08's user avatar
0 votes
0 answers
16 views

Update/ recalibrate XG Boost, Random Forest, GLM models for external validation

I have created XG Boost, Random Forest and GLM models for classification of a binary outcome and now I want to externally validate the models on a different population of over 5000 subjects. I have ...
mmo's user avatar
  • 11
0 votes
1 answer
72 views

Do uncalibrated "probability" predictions satisfy Kolmogorov's axioms?

Let's say we have some binary variable of interest and fit a model to predict the probability of the two classes, say a logistic regression or a "classification" neural network. This model ...
Dave's user avatar
  • 58k
1 vote
0 answers
96 views

Is perfect isotonic probability calibration realistic?

I work with a labelled tabular dataset of about 1 million observations, with the target being binary. The dataset is heavily imbalanced - about 0.5% positive class. I have trained a gradient boosting ...
StrLdn's user avatar
  • 11
1 vote
0 answers
58 views

Understanding a calibration plot for lightGBM binary classifier

I wanted to assess the performance of my lightGBM classifier using a calibration plot. If I understood correctly, a calibration plot visualizes the alignment between the predicted probabilities by the ...
Programming Noob's user avatar
1 vote
0 answers
48 views

Assessing uncertainty calibration in regression using the CDF

I have a labelled data set with $n$ data points $(x_i, y_i)$ with $x_i \in \mathbb{R}^k$ and $y_i \in \mathbb{R}$ and I trained a model $f: \mathbb{R}^k \to \mathbb{R} \times \mathbb{R}^+$ on some of ...
PascalIv's user avatar
  • 799
1 vote
0 answers
26 views

How to get standard error from constrained optimization problem in R?

Can I get standard error from a constrained optimization problem in R? I have calculated transition probabilities. Now I am trying to calibrate it. Using these transition probabilities I have ...
Md. Zubab Ibne Moid's user avatar
0 votes
1 answer
76 views

Model performance with multiply imputed data

I would like to know how to do calibration plot with Hosmer-lemeshow test and AUC for ROC curve after multiple imputation in R. I build one prediction model and tried to do model performance but ...
Haruka Hayashi's user avatar
1 vote
0 answers
62 views

Model calibration in overfitted models

Why in Shrinkage, due to an overfitted prediction model, do we tend to overestimate risk for "high risk" subjects and to underestimate risk for "low risk" subjects ? Intuitively I ...
vixxovs's user avatar
  • 35
0 votes
1 answer
84 views

Optimizing a threshold value on a dependent metric using a classifier trained to optimize a threshold-independent metric

Is it a reasonable approach to train a probabilities classifier by optimizing a threshold-independent metric such as AUC, and then using the trained classifier to calibrate the decision threshold ...
Amit S's user avatar
  • 115
0 votes
0 answers
73 views

(Un)Calibrated Logistic Regression Fit

I'm fitting a logistic regression on a large dataset (n=89260, 17 predictors) with a class imbalance (1% positive class). I've tried to follow Dr. Harrell's teachings so I fit my full pre-specified ...
Daniel Nunes's user avatar
0 votes
0 answers
44 views

Calibration plot without binning predictions

Similar to ths question I would like to know how to create a calibration curve without binning my predictions. What makes my situation different, is that I'm using icenReg for my interval-censored ...
Wojty's user avatar
  • 63
0 votes
1 answer
80 views

How does someone achieve a desired confidence / accuracy when measuring using uncalibrated instrument?

I have an instrument that measures a value. It is only possible to measure the value once i.e. the experiment can't be repeated (think recording a car's speed as it drives past). The instrument is not ...
Chuck's user avatar
  • 81
0 votes
0 answers
13 views

measure Discrimination and Calibration in case-control study

When I adopt a case-control study for risk prediction models of CC, how I can measure Discrimination and Calibration?
user388058's user avatar
0 votes
0 answers
51 views

At what point during model development can model calibration be applied?

I have been working on prediction models in R studio based on a rather small data set. There is a total of ~ 1200 cases with 150 to 400 positive cases depending on which of the different outcomes is ...
mmo's user avatar
  • 11
0 votes
0 answers
18 views

Question on Calibrated Structured Prediction (Kuleshov, 2015)

I have several questions regarding equation (2) from the paper: Calibrated Structured Prediction (Kuleshov, 2015). I believe most of my questions stem from not understanding the quantity: $T(x)=\...
confused_grad_student's user avatar
1 vote
0 answers
91 views

Softmax Response vs MC Dropout for Uncertainty Estimation [closed]

Some papers I see take the uncertainty estimation of a prediction as simply its softmax/sigmoid output, whereas some papers will use techniques such as MC Dropout and calculate the variance across the ...
user8903610's user avatar
1 vote
0 answers
385 views

Creating and interpreting calibration plots for several models with a binary outcome

I have made several models (RF, XGB and GLM) to predict a binary outcome and they all achieved an AUC of approximately 0.8 and Brier scores 0.1-0.15. Test set is fairly small (n= 350), cases with ...
mmo's user avatar
  • 11
0 votes
0 answers
214 views

AUC values of training and cross-validation are lower than AUC values of test set

I am training a Full model (logistic regression) and a few different models (LASSO, Elastic net, CART, random forest) to predict a certain clinical outcome. I split my data into training and test sets ...
SAN's user avatar
  • 71
3 votes
1 answer
2k views

Create calibration plot in R with vectors of predicted and observed values

I am currently working on a project regarding the external validation of a logistic regression model for binary classification. I would like to create a calibration plot and compute the calibration-in-...
Yan Linxuan's user avatar
4 votes
3 answers
521 views

Low p-value, but many trials

I am not well-versed in statistics, so please bear with me. I will try to explain things to the best of my knowledge. I have some hardware that produces a signal, this signal has an offset and I want ...
user17004502's user avatar
0 votes
0 answers
10 views

Recalibrate scores across departments?

I'm looking for some assistance in the procedures to use to recalibrate subjective scores of staff performance at our university. Say we have 10 different departments, each with 10 faculty, and each ...
user380421's user avatar
0 votes
0 answers
55 views

Model for probability threshold selection, how to make differentiable loss?

I have an object detection model that outputs bounding boxes and probabilities associated with them. The probability expresses the chance that the object found is a tumor. I have noticed that the ...
Slajni's user avatar
  • 45
2 votes
2 answers
289 views

Should sum of sample weights always be equal to the target population

I have a simple question regarding sample weighting: From what I understand, sample weights are used to reduce or eliminate potential biases arising from the differences between the selected sample ...
Willi Zhang's user avatar
3 votes
1 answer
523 views

Why do we need separate data for probability calibration?

Why do we need separate data for probability calibration? Scikit learn documentation says: The samples that are used to fit the calibrator should not be the same samples used to fit the classifier, as ...
Glue's user avatar
  • 443
1 vote
0 answers
32 views

Neural Networks Miscalibration Measure

I have read these two papers related to the neural network miscalibration problem: "On Calibration of Modern Neural Networks" and "Multivariate Confidence Calibration for Object ...
alimagadovk's user avatar
1 vote
1 answer
45 views

Does probability calibration descrease model prediction variance?

Does probability calibration decrease model prediction variance? Example: Let's say we have a classifier that is a mail spam detector. It outputs a score between 0-1 to quantify how likely a given ...
Glue's user avatar
  • 443
2 votes
0 answers
68 views

Bayesian calibration of computer simulations - Likelihood function calculation

I am starting to study Bayesian calibrations of computer models. I am not a statistician and just starting to learn so bear with me if I do not use the correct terminology. The general approach is ...
Rojj's user avatar
  • 121
2 votes
0 answers
279 views

Does scikit-learn support plotting calibration curve for multiclass classifier?

I have trained a multi class classifier and calibrated the classified using scikit-learn library's CalibratedClassifierCV class. To check how well the probabilities are calibrated, I tried plotting ...
yodasoda18's user avatar
0 votes
0 answers
51 views

Should I address the imbalance when using CalibratedClassifierCV?

Im using RandomForestClassifier and XGBClassifier with an imbalanced dataset, 1:2 ratio more or less, 1 being the most prevalent class. My procedure is the following: Use StratifiedKFold to get ...
Martin Gorosito's user avatar
0 votes
0 answers
35 views

Post training calibration of BNN and dataset split

since i am working with a small dataset (1048 rows) with much of the data concentrated in the region for which i have no interest, i was wondering if for Bayesian neural networks it is necessary to ...
Alucard's user avatar
  • 325
3 votes
1 answer
2k views

Probability Calibration for Highly Imbalanced Binary Classification

I am working on a binary classification problem on a highly imbalanced dataset (1:100) where model probabilities are important for the use case and need to be well calibrated to best represent true ...
Jake Niederer's user avatar
1 vote
1 answer
52 views

Is the true conditional $P(Y \mid X)$ calibrated?

I am trying to get an intuitive understanding of the concept of calibration. Definitions first. Consider a data distribution $P(X, Y)$ over binary labels $Y$, and a probabilistic classifier which ...
usual me's user avatar
  • 1,167
1 vote
0 answers
174 views

How do you do decision threshold tuning when doing k-fold cross validation?

I'm training a binary classifier for disease detection. Because of my small amount of data (~1000 datapoints, 10% positive, 90% negative), I've realized that doing an 80-20 train-test split produces ...
jimbo's user avatar
  • 11
1 vote
1 answer
495 views

Correct way to interpret sklearn's calibration error and generate a numeric calibration loss/score

I have been using sklearn's CalibrationDisplay and think it is pretty cool. One thing I am wondering, though, is how I could potentially take that curve and make it an interpretable score. For example,...
bismo's user avatar
  • 121
3 votes
1 answer
70 views

Probability Calibration, what causes simple linear models like logistic regression to be over-confident and diverge from the true class probabilities?

I've recently studied probability calibration and have investigated many examples and revisited old models of mine to find that they are all poorly calibrated. The idea of stacking also has been cast ...
Oliver's user avatar
  • 290
0 votes
1 answer
131 views

Cross-validation curves in Cox model with time dependant covariate

I need to make cross-validation curves to check the validity of a Cox model. So the data are divided in training and test. The coefficients are estimated on the training and the predictor linears are ...
Flora Grappelli's user avatar
9 votes
2 answers
791 views

How does the Brier Score break down to (Reliability - Resolution + Uncertainty)?

The Wikipedia page states this in the decompositions section, and it is also stated in an older paper I have never been able to understand these explanations, and I wonder what I am missing and if ...
Joff's user avatar
  • 812
1 vote
1 answer
68 views

Is it ok to widen a prior during an MCMC which did not converge yet?

I am calibrating parameters of a process model. The runtime of the model is high and the calibration already ran for more than two weeks with many cores on a HPC. After almost 150k iterations I ...
Hans Jürgen's user avatar
1 vote
0 answers
18 views

Determine drift

I would like to determine potential drift in the data/device. Is there a general acknowledged procedure and/or method to do so? I know how to visually "detect" a drift but I would like to ...
Ben's user avatar
  • 3,153
2 votes
1 answer
68 views

When developing models for effect estimation, is it advisable to perform internal validation and calibration using resampling methods? #rms

In predictive modelling, it is useful to perform resampling methods to assess performance e.g. validate() and calibrate() in the rms package in R. Is it useful / advisable to perform such methods when ...
user167591's user avatar
3 votes
1 answer
231 views

How damaging to the analysis would it be to run probability validation (`rms::val.prob`) when calibration (`rms::calibrate`) is the correct action?

If I make a model that predicts probabilities (e.g., logistic regression or a neural network), I would like it to have the property that, when it predicts a probability of $p$, the event happens about ...
Dave's user avatar
  • 58k
1 vote
0 answers
79 views

what does it mean when the NLL becomes negative?

i was comparing the results of 3 different techniques for regression task( Deep ensembles, variational inference and concrete dropout) and i got these results from the table looks like everything is ...
Alucard's user avatar
  • 325
2 votes
0 answers
206 views

How to assess calibration of probability distribution for a multiclass model?

I have a multiclass classifier (boosting model), and my goal is to have a good approximation of the actual distribution to the classes given my feature values. I.e. suppose I have features $X$, and ...
Lys's user avatar
  • 56

1
2 3 4 5
7