Questions tagged [calibration]
Calibration can refer to predicting an explanatory variable from an observation of the dependent variable; to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc.
150
questions
0
votes
2answers
15 views
What is probability estimates calibration?
I am going through this where the author is talking about calibration of prediction estimates of a neural network. I tried to find an intuitive explanation of what, why and how regarding calibration ...
0
votes
0answers
10 views
Quantifying a model's probability calibration
Just wondering if there is a standard way of quantifying calibration curves for classification similar to those here: scikit-learn.
1
vote
1answer
12 views
Merging two model results
My aim is to predict a binary output, given instances X and associated binary outputs y. I built a first classifier out-putting scores (not calibrated in probabilities). However, I identified in my ...
1
vote
1answer
36 views
Calibration curve of XGBoost for binary classification
I'm working on a binary classification problem, with imbalanced classes (10:1). Since for binary classification, the objective function of XGBoost is ...
0
votes
0answers
38 views
Regression and calibration/inverse regression - the same?
In principle, I've a simple task:
There are some physical quantities (temperature, pressure and molecule number) measured by a sensor and I want to use them to regress the concenctration of a gas. I ...
0
votes
1answer
44 views
Follow up question to: Why does logistic regression generate well-calibrated models? [duplicate]
I've read all answers and comments here: Why does logistic regression produce well-calibrated models?
but still not clear about the answer.
Can someone please elaborate why the following equation ...
0
votes
0answers
6 views
Test set achieving better calibration (Lower Hosmer Lemeshow chi square value) than train set
I am trying to compare different ML models on a data set with binary outcome. Discriminating ability is scored using AUROC and calibration using HL test. The outcome of interest is only 10% of the ...
1
vote
0answers
15 views
How to compare predicted accuracy and actual accuracy? [duplicate]
Consider a classifier that, given an input vector ${\bf x}$ outputs both a prediction $y'$ whose accuracy ($a \in \{0, 1\}$) can be measured, as well as a predicted accuracy that corresponds to the ...
0
votes
0answers
27 views
Why we use calibration of Machine Learning models?
From the different websites I got to know how to perform calibration of models. But can anyone tell me the reason behind performing calibration of machine learning models?
2
votes
0answers
26 views
How to aggregate calibration curves which were created in cross validation?
When looking into Scikit's CalibratedClassifierCV I noticed that the object needs to keep multiple calibrated classifiers in memory to average the results in real time.
I understand that these ...
2
votes
0answers
26 views
Self-calibrate sensor measurements with linear regression
I have measurements of 3 sensors (A, B, C) over variable W. Reproducible code.
The sensors may be wrongly calibrated so before analysing the data I correct the values for each sensor. The correction ...
1
vote
1answer
84 views
Logistic regression produces well calibrated models. Is that true for neural nets trained in batches?
This is an earlier discussion about LR producing well calibrated models:
Some people equate neural net based prediction models (even deep NN or deep+sparse NN) to be equivalent to logistic ...
2
votes
0answers
8 views
Uncertainty in calibration/curve-fitting parameters
Let me preface with saying I have an idea of a solution, but I am interested in other ones
I am interested in the a way to quantify the uncertainty in a calibration/curve-fit parameter. For all ...
1
vote
0answers
108 views
What does the logistic calibration line represent on the calibration plot made by val.plot function (from rms package in R)?
I've been trying to make calibration plots/curves for a logistic regression model in R to mimic what I'm doing in a statistics course--but the class uses SAS with proc sgplot.
I think I've figured it ...
0
votes
1answer
46 views
Re-calibrating Intercept on logistic regression models for unbalance data
I have data-set that I’m modelling using logistic regression as land.cover~H1+H2+H3+H4+H6+H8+H14. My response and categorical variables are binary. However the number of 0 and 1 in my response ...
0
votes
0answers
28 views
Weights in “visited website” survey
I have a dataset with the browsing history of people in a country with age and gender, my goal is to give some information regarding some websites like :
the market shares between several website ...
0
votes
0answers
98 views
Survey weights: calibration with continuous variables
I'm using the R survey package and trying to run the calibrate function. My setup:
For each individual I have age, gender, ...
1
vote
0answers
47 views
Calibration curve for mixed model (logistic) [closed]
I did some searching using the CV search function and was unable to locate any information on R packages or general approach to creating a calibration curve with a bootstrapped curve overlay (similar ...
8
votes
2answers
399 views
Why is logistic regression well calibrated, and how to ruin its calibration?
In the scikit learn documents on probability calibration they compare logistic regression with other methods and remark that random forest is less well calibrated than logistic regression.
Why is ...
2
votes
0answers
44 views
Calibrating probabilities of a binary classifier when class prior is unknown
Is it possible to calibrate the probabilities of a binary classifier when the class priors are unknown?
In cases where the data is obtained with selection bias (i.e. more positives than negatives in ...
1
vote
0answers
103 views
How to validate(with sample-split data) and calibrate Cox model with time-dependent covaraites?
I am building 2 cox models:
Without time-dependent covariates
With time-dependent covariates.
1.The first model (without time-dependent variables) as specified below in R works fine and I have no ...
1
vote
2answers
241 views
Linear predictor from coefficients of Cox PH model
I need to calculate the linear predictor of a Cox PH model by hand.
I can get continuous and binary variables to match the output of predict.coxph (specifying 'lp') but I can't seem to figure out how ...
0
votes
0answers
83 views
Calculation of RMSEC (Calibration Error), Which data should be used?
I got a bit confused about RMSEC but I got how RMSECV and RMSEP are calculated and their meaning, though.
I am working based on 2 papers which summarize their results with RMSEC, RMSECV1, RMSECV2 and ...
0
votes
0answers
28 views
Sample Size Adjusted Metric for Calibration Curve
I'm creating a model that provides a predicted probability of an event happening. I am using calibration curves that plot predicted probability (x-axis) versus the actual event frequency (y-axis). I ...
3
votes
1answer
64 views
Measuring uncertainty of a fitted calibration curve
My question is: how can you compute the prediction interval of a calibration curve, just like you might for any other regression model.
A calibration curve maps estimated probabilities to empirical ...
3
votes
0answers
568 views
Probability calibration from LightGBM model with class imbalance
I've made a binary classification model using LightGBM. The dataset was fairly imbalnced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. ...
3
votes
0answers
234 views
Calibration of penalized (LASSO or ELasticNet) logistic regression models
I would be very grateful for any help me with the following general query regarding calibration of penalized models with a binary outcome.
I would like my prediction model to be calibrated (mean ...
2
votes
1answer
183 views
why is calibration of my logistic regression s shaped?
I am simulating data to compare real and predicted probabilities from logistic regression like this:
...
0
votes
1answer
23 views
Similarity between variables when using calibration curves
I'm reading through a text that is explaining calibration curves, and the following description is provided:
To be well-calibrated, the probabilities must effectively reflect the true likelihood of ...
1
vote
0answers
43 views
Boosted Regression Trees: Zero discrimination and calibration scores
I am using boosted regression trees using the gbm() and dismo() packages. When I run my models I get values of zero for discrimination and calibration in the $cv.statistics
What would be the reason ...
6
votes
1answer
528 views
Does Regularized Logistic Regression Produce Calibrated Results?
It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow ...
1
vote
1answer
558 views
Probability calibration metric for multiclass classifier
A machine learning classifier can be calibrated so that when the probability that datapoint i is of class A is 0.6, this is true 60% of the time.
In the binary class setting, this can be visualised ...
0
votes
1answer
305 views
Finding an optimal class probability threshold for SVM
I've got an imbalanced data set on which I'm training an SVM using cross validation. I'd like to find the optimal class probability threshold that maximizes the F measure.
I've tried doing this by ...
0
votes
1answer
44 views
Calibration of an individual-based model of an epidemic
I am currently developing an individual-based (or agent-based) mathematical model (IBM) of an epidemic. I want to calibrate the transmission parameters in my IBM to match empirical data (epidemic ...
1
vote
0answers
123 views
R caret calibration function
I'm studying about calibration techniques and I'd like to use the CARET calibration function to examine the quality of my classifiers' probabilities.
I read the relevant documentation, however I don'...
2
votes
0answers
492 views
Create calibration plot with error bars for logistic regression model
I would like to create a calibration plot for a logistic regression model along with 95% confidence intervals for the mean predicted probability in each bin. The plot I'm after along with the code to ...
3
votes
0answers
86 views
inverse.predict chemcal package
I have noticed that the inverse.predict function (chemCal package) does not take into account all the degrees of freedom of the ...
2
votes
0answers
285 views
Calibration Curve v. Linear Dynamic Range [closed]
We have a piece of analytical chemistry equipment (a mass spectrometer for those who care), that has the capacity to generate a "Response value" (Area under the curve) that correlates to the amount of ...
2
votes
0answers
287 views
Interpretation of this calibration curve on prediction models
I have several prediction models that I present in this calibration curve using R with the package "riskRegression".
In this study, the participants are community-dwelling older persons followed up ...
1
vote
0answers
14 views
Detecting outlying instances from probability estimates
Say I have a binary classification model that is able to produce (posterior) probability estimates. I check whether these probabilities are calibrated and I evaluate using a metric that describes its ...
8
votes
1answer
670 views
How to estimate a calibration curve with bootstrap (R)
Question: I have fitted a probabilistic model (bayesian network) for modeling a binary outcome variable. I would like to create a high-resolution calibration plot (e.g. spline) corrected for ...
0
votes
1answer
330 views
How to plot the calibration curve for an ordinal logistic regression model applied to a test sample?
I'm doing a validation study of an ordinal logistic regression model that was made with the lrm function of the rms package in R. How can I plot the calibration curve for the model when applied to new ...
1
vote
0answers
32 views
Goodness of Fit (weighted least squared) using variance instead of standard deviation
Is goodness of fit measured by weighted least square sometimes done by using the variance instead of the standard deviation in the denominator?
What I mean is the following. Lets consider the ...
2
votes
1answer
43 views
Correlation of model and experimental results
I am trying to find a scientifically accepted method to compare my model predictions with experimental values. Here, my model predicts the concentration of DNA as a function of time. DNA concentration ...
2
votes
0answers
128 views
Is there a maximum value for AUC for a given value of calibration?
Is there an upper boundary for the AUC of a risk model that is dependent on both the model calibration and the frequency of the outcome under examination?
Cook suggests that high values for ...
0
votes
1answer
31 views
Impact of the model error in numerical model prediction
I'm having trouble wrapping my head around characterizing model prediction uncertainties in the case of a calibrated numerical model. Let's assume we are trying to calibrate a set of input parameters ...
0
votes
1answer
105 views
Best way to check if two variables are correlated
(a) I have two variables that I am looking at: Visceral fat levels using MRI, and the visceral fat rating as shown by bioelectrical impedance (BIA) scales. The units for these two variables are ...
1
vote
0answers
204 views
R: Cox with time-dependent covariates Validation (discrimination/calibration) package [closed]
I am working on a dataset with multiple measures for each subject (id) and thus time-dependent covariates (measurement_value). I am interested in the "status" outcome. Some subjects do experience the ...
1
vote
1answer
142 views
How to match curves or measure the similarity between both?
Firstly, I'm not sure whether this is the correct site but I also assume the question is not too different:
Assume there are two curves: One is considered as a reference curve and the other is just ...
1
vote
0answers
18 views
Calibration of parametric multi-agent simulation to be consistent with emergent stylized facts
I've created a market simulation with a few different types of agents that trade with each other based on simple rules that sample from parametrized random distributions.
I can choose reasonable ...
