# Questions tagged [calibration]

Calibration can refer to predicting an explanatory variable from an observation of the dependent variable; to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc.

150 questions
Filter by
Sorted by
Tagged with
15 views

### What is probability estimates calibration?

I am going through this where the author is talking about calibration of prediction estimates of a neural network. I tried to find an intuitive explanation of what, why and how regarding calibration ...
10 views

### Quantifying a model's probability calibration

Just wondering if there is a standard way of quantifying calibration curves for classification similar to those here: scikit-learn.
12 views

### Merging two model results

My aim is to predict a binary output, given instances X and associated binary outputs y. I built a first classifier out-putting scores (not calibrated in probabilities). However, I identified in my ...
36 views

### Calibration curve of XGBoost for binary classification

I'm working on a binary classification problem, with imbalanced classes (10:1). Since for binary classification, the objective function of XGBoost is ...
38 views

### Regression and calibration/inverse regression - the same?

In principle, I've a simple task: There are some physical quantities (temperature, pressure and molecule number) measured by a sensor and I want to use them to regress the concenctration of a gas. I ...
44 views

6 views

### Test set achieving better calibration (Lower Hosmer Lemeshow chi square value) than train set

I am trying to compare different ML models on a data set with binary outcome. Discriminating ability is scored using AUROC and calibration using HL test. The outcome of interest is only 10% of the ...
15 views

### How to compare predicted accuracy and actual accuracy? [duplicate]

Consider a classifier that, given an input vector ${\bf x}$ outputs both a prediction $y'$ whose accuracy ($a \in \{0, 1\}$) can be measured, as well as a predicted accuracy that corresponds to the ...
27 views

### Why we use calibration of Machine Learning models?

From the different websites I got to know how to perform calibration of models. But can anyone tell me the reason behind performing calibration of machine learning models?
26 views

### How to aggregate calibration curves which were created in cross validation?

When looking into Scikit's CalibratedClassifierCV I noticed that the object needs to keep multiple calibrated classifiers in memory to average the results in real time. I understand that these ...
26 views

### Self-calibrate sensor measurements with linear regression

I have measurements of 3 sensors (A, B, C) over variable W. Reproducible code. The sensors may be wrongly calibrated so before analysing the data I correct the values for each sensor. The correction ...
84 views

### Logistic regression produces well calibrated models. Is that true for neural nets trained in batches?

This is an earlier discussion about LR producing well calibrated models: Some people equate neural net based prediction models (even deep NN or deep+sparse NN) to be equivalent to logistic ...
8 views

### Uncertainty in calibration/curve-fitting parameters

Let me preface with saying I have an idea of a solution, but I am interested in other ones I am interested in the a way to quantify the uncertainty in a calibration/curve-fit parameter. For all ...
108 views

### What does the logistic calibration line represent on the calibration plot made by val.plot function (from rms package in R)?

I've been trying to make calibration plots/curves for a logistic regression model in R to mimic what I'm doing in a statistics course--but the class uses SAS with proc sgplot. I think I've figured it ...
46 views

### Re-calibrating Intercept on logistic regression models for unbalance data

I have data-set that I’m modelling using logistic regression as land.cover~H1+H2+H3+H4+H6+H8+H14. My response and categorical variables are binary. However the number of 0 and 1 in my response ...
28 views

### Weights in “visited website” survey

I have a dataset with the browsing history of people in a country with age and gender, my goal is to give some information regarding some websites like : the market shares between several website ...
98 views

### Survey weights: calibration with continuous variables

I'm using the R survey package and trying to run the calibrate function. My setup: For each individual I have age, gender, ...
47 views

### Calibration curve for mixed model (logistic) [closed]

I did some searching using the CV search function and was unable to locate any information on R packages or general approach to creating a calibration curve with a bootstrapped curve overlay (similar ...
399 views

### Why is logistic regression well calibrated, and how to ruin its calibration?

In the scikit learn documents on probability calibration they compare logistic regression with other methods and remark that random forest is less well calibrated than logistic regression. Why is ...
44 views

### Calibrating probabilities of a binary classifier when class prior is unknown

Is it possible to calibrate the probabilities of a binary classifier when the class priors are unknown? In cases where the data is obtained with selection bias (i.e. more positives than negatives in ...
103 views

### How to validate(with sample-split data) and calibrate Cox model with time-dependent covaraites?

I am building 2 cox models: Without time-dependent covariates With time-dependent covariates. 1.The first model (without time-dependent variables) as specified below in R works fine and I have no ...
241 views

### Linear predictor from coefficients of Cox PH model

I need to calculate the linear predictor of a Cox PH model by hand. I can get continuous and binary variables to match the output of predict.coxph (specifying 'lp') but I can't seem to figure out how ...
83 views

### Calculation of RMSEC (Calibration Error), Which data should be used?

I got a bit confused about RMSEC but I got how RMSECV and RMSEP are calculated and their meaning, though. I am working based on 2 papers which summarize their results with RMSEC, RMSECV1, RMSECV2 and ...
28 views

### Sample Size Adjusted Metric for Calibration Curve

I'm creating a model that provides a predicted probability of an event happening. I am using calibration curves that plot predicted probability (x-axis) versus the actual event frequency (y-axis). I ...
64 views

### Measuring uncertainty of a fitted calibration curve

My question is: how can you compute the prediction interval of a calibration curve, just like you might for any other regression model. A calibration curve maps estimated probabilities to empirical ...
568 views

### Probability calibration from LightGBM model with class imbalance

I've made a binary classification model using LightGBM. The dataset was fairly imbalnced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. ...
234 views

### Calibration of penalized (LASSO or ELasticNet) logistic regression models

I would be very grateful for any help me with the following general query regarding calibration of penalized models with a binary outcome. I would like my prediction model to be calibrated (mean ...
183 views

### why is calibration of my logistic regression s shaped?

I am simulating data to compare real and predicted probabilities from logistic regression like this: ...
23 views

### Similarity between variables when using calibration curves

I'm reading through a text that is explaining calibration curves, and the following description is provided: To be well-calibrated, the probabilities must effectively reflect the true likelihood of ...
43 views

### Boosted Regression Trees: Zero discrimination and calibration scores

I am using boosted regression trees using the gbm() and dismo() packages. When I run my models I get values of zero for discrimination and calibration in the \$cv.statistics What would be the reason ...
528 views

### Does Regularized Logistic Regression Produce Calibrated Results?

It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow ...
558 views

### Probability calibration metric for multiclass classifier

A machine learning classifier can be calibrated so that when the probability that datapoint i is of class A is 0.6, this is true 60% of the time. In the binary class setting, this can be visualised ...
305 views

### Finding an optimal class probability threshold for SVM

I've got an imbalanced data set on which I'm training an SVM using cross validation. I'd like to find the optimal class probability threshold that maximizes the F measure. I've tried doing this by ...
44 views

### Calibration of an individual-based model of an epidemic

I am currently developing an individual-based (or agent-based) mathematical model (IBM) of an epidemic. I want to calibrate the transmission parameters in my IBM to match empirical data (epidemic ...
123 views

### R caret calibration function

I'm studying about calibration techniques and I'd like to use the CARET calibration function to examine the quality of my classifiers' probabilities. I read the relevant documentation, however I don'...
492 views

### Create calibration plot with error bars for logistic regression model

I would like to create a calibration plot for a logistic regression model along with 95% confidence intervals for the mean predicted probability in each bin. The plot I'm after along with the code to ...
86 views

### inverse.predict chemcal package

I have noticed that the inverse.predict function (chemCal package) does not take into account all the degrees of freedom of the ...
285 views

### Calibration Curve v. Linear Dynamic Range [closed]

We have a piece of analytical chemistry equipment (a mass spectrometer for those who care), that has the capacity to generate a "Response value" (Area under the curve) that correlates to the amount of ...
287 views

### Interpretation of this calibration curve on prediction models

I have several prediction models that I present in this calibration curve using R with the package "riskRegression". In this study, the participants are community-dwelling older persons followed up ...
14 views

### Detecting outlying instances from probability estimates

Say I have a binary classification model that is able to produce (posterior) probability estimates. I check whether these probabilities are calibrated and I evaluate using a metric that describes its ...
670 views

### How to estimate a calibration curve with bootstrap (R)

Question: I have fitted a probabilistic model (bayesian network) for modeling a binary outcome variable. I would like to create a high-resolution calibration plot (e.g. spline) corrected for ...
330 views

### How to plot the calibration curve for an ordinal logistic regression model applied to a test sample?

I'm doing a validation study of an ordinal logistic regression model that was made with the lrm function of the rms package in R. How can I plot the calibration curve for the model when applied to new ...
32 views

### Goodness of Fit (weighted least squared) using variance instead of standard deviation

Is goodness of fit measured by weighted least square sometimes done by using the variance instead of the standard deviation in the denominator? What I mean is the following. Lets consider the ...
43 views

### Correlation of model and experimental results

I am trying to find a scientifically accepted method to compare my model predictions with experimental values. Here, my model predicts the concentration of DNA as a function of time. DNA concentration ...
128 views

### Is there a maximum value for AUC for a given value of calibration?

Is there an upper boundary for the AUC of a risk model that is dependent on both the model calibration and the frequency of the outcome under examination? Cook suggests that high values for ...
31 views

### Impact of the model error in numerical model prediction

I'm having trouble wrapping my head around characterizing model prediction uncertainties in the case of a calibrated numerical model. Let's assume we are trying to calibrate a set of input parameters ...
105 views

### Best way to check if two variables are correlated

(a) I have two variables that I am looking at: Visceral fat levels using MRI, and the visceral fat rating as shown by bioelectrical impedance (BIA) scales. The units for these two variables are ...
204 views

### R: Cox with time-dependent covariates Validation (discrimination/calibration) package [closed]

I am working on a dataset with multiple measures for each subject (id) and thus time-dependent covariates (measurement_value). I am interested in the "status" outcome. Some subjects do experience the ...