Questions tagged [calibration]
Calibration can refer to predicting an explanatory variable from an observation of the dependent variable; to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc.
1
vote
0answers
13 views
How to compare predicted accuracy and actual accuracy? [duplicate]
Consider a classifier that, given an input vector ${\bf x}$ outputs both a prediction $y'$ whose accuracy ($a \in \{0, 1\}$) can be measured, as well as a predicted accuracy that corresponds to the ...
0
votes
0answers
19 views
Why we use calibration of Machine Learning models?
From the different websites I got to know how to perform calibration of models. But can anyone tell me the reason behind performing calibration of machine learning models?
2
votes
0answers
17 views
How to aggregate calibration curves which were created in cross validation?
When looking into Scikit's CalibratedClassifierCV I noticed that the object needs to keep multiple calibrated classifiers in memory to average the results in real time.
I understand that these ...
2
votes
0answers
21 views
Self-calibrate sensor measurements with linear regression
I have measurements of 3 sensors (A, B, C) over variable W. Reproducible code.
The sensors may be wrongly calibrated so before analysing the data I correct the values for each sensor. The correction ...
0
votes
0answers
28 views
we say that logistic regression produces well calibrated models, is that still true for neural nets trained in batches?
This link has previous discussion about LR producing well calibrated models:
Why does logistic regression produce well-calibrated models?
Some people equate neural net based prediction models (even ...
2
votes
0answers
6 views
Uncertainty in calibration/curve-fitting parameters
Let me preface with saying I have an idea of a solution, but I am interested in other ones
I am interested in the a way to quantify the uncertainty in a calibration/curve-fit parameter. For all ...
1
vote
0answers
51 views
What does the logistic calibration line represent on the calibration plot made by val.plot function (from rms package in R)?
I've been trying to make calibration plots/curves for a logistic regression model in R to mimic what I'm doing in a statistics course--but the class uses SAS with proc sgplot.
I think I've figured it ...
0
votes
1answer
29 views
Re-calibrating Intercept on logistic regression models for unbalance data
I have data-set that I’m modelling using logistic regression as land.cover~H1+H2+H3+H4+H6+H8+H14. My response and categorical variables are binary. However the number of 0 and 1 in my response ...
0
votes
0answers
27 views
Weights in “visited website” survey
I have a dataset with the browsing history of people in a country with age and gender, my goal is to give some information regarding some websites like :
the market shares between several website ...
0
votes
0answers
65 views
Survey weights: calibration with continuous variables
I'm using the R survey package and trying to run the calibrate function. My setup:
For each individual I have age, gender, ...
1
vote
0answers
40 views
Calibration curve for mixed model (logistic) [closed]
I did some searching using the CV search function and was unable to locate any information on R packages or general approach to creating a calibration curve with a bootstrapped curve overlay (similar ...
8
votes
2answers
296 views
Why is logistic regression well calibrated, and how to ruin its calibration?
In the scikit learn documents on probability calibration they compare logistic regression with other methods and remark that random forest is less well calibrated than logistic regression.
Why is ...
2
votes
0answers
43 views
Calibrating probabilities of a binary classifier when class prior is unknown
Is it possible to calibrate the probabilities of a binary classifier when the class priors are unknown?
In cases where the data is obtained with selection bias (i.e. more positives than negatives in ...
1
vote
0answers
69 views
How to validate(with sample-split data) and calibrate Cox model with time-dependent covaraites?
I am building 2 cox models:
Without time-dependent covariates
With time-dependent covariates.
1.The first model (without time-dependent variables) as specified below in R works fine and I have no ...
0
votes
2answers
166 views
Linear predictor from coefficients of Cox PH model
I need to calculate the linear predictor of a Cox PH model by hand.
I can get continuous and binary variables to match the output of predict.coxph (specifying 'lp') but I can't seem to figure out how ...
0
votes
0answers
70 views
Calculation of RMSEC (Calibration Error), Which data should be used?
I got a bit confused about RMSEC but I got how RMSECV and RMSEP are calculated and their meaning, though.
I am working based on 2 papers which summarize their results with RMSEC, RMSECV1, RMSECV2 and ...
0
votes
0answers
25 views
Sample Size Adjusted Metric for Calibration Curve
I'm creating a model that provides a predicted probability of an event happening. I am using calibration curves that plot predicted probability (x-axis) versus the actual event frequency (y-axis). I ...
4
votes
1answer
53 views
Measuring uncertainty of a fitted calibration curve
My question is: how can you compute the prediction interval of a calibration curve, just like you might for any other regression model.
A calibration curve maps estimated probabilities to empirical ...
2
votes
0answers
416 views
Probability calibration from LightGBM model with class imbalance
I've made a binary classification model using LightGBM. The dataset was fairly imbalnced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. ...
3
votes
0answers
189 views
Calibration of penalized (LASSO or ELasticNet) logistic regression models
I would be very grateful for any help me with the following general query regarding calibration of penalized models with a binary outcome.
I would like my prediction model to be calibrated (mean ...
2
votes
1answer
138 views
why is calibration of my logistic regression s shaped?
I am simulating data to compare real and predicted probabilities from logistic regression like this:
...
0
votes
1answer
23 views
Similarity between variables when using calibration curves
I'm reading through a text that is explaining calibration curves, and the following description is provided:
To be well-calibrated, the probabilities must effectively reflect the true likelihood of ...
1
vote
0answers
36 views
Boosted Regression Trees: Zero discrimination and calibration scores
I am using boosted regression trees using the gbm() and dismo() packages. When I run my models I get values of zero for discrimination and calibration in the $cv.statistics
What would be the reason ...
6
votes
1answer
478 views
Does Regularized Logistic Regression Produce Calibrated Results?
It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow ...
1
vote
1answer
451 views
Probability calibration metric for multiclass classifier
A machine learning classifier can be calibrated so that when the probability that datapoint i is of class A is 0.6, this is true 60% of the time.
In the binary class setting, this can be visualised ...
0
votes
1answer
268 views
Finding an optimal class probability threshold for SVM
I've got an imbalanced data set on which I'm training an SVM using cross validation. I'd like to find the optimal class probability threshold that maximizes the F measure.
I've tried doing this by ...
0
votes
1answer
43 views
Calibration of an individual-based model of an epidemic
I am currently developing an individual-based (or agent-based) mathematical model (IBM) of an epidemic. I want to calibrate the transmission parameters in my IBM to match empirical data (epidemic ...
1
vote
0answers
109 views
R caret calibration function
I'm studying about calibration techniques and I'd like to use the CARET calibration function to examine the quality of my classifiers' probabilities.
I read the relevant documentation, however I don'...
-1
votes
0answers
17 views
Metric to determine distance of good pixels to a target value in an image with many defective pixels
An xray source emits radiation with strength being a function of some variables, one is kv. The xray strength is supposed to increase monotonically as kv increases.
Then an xray detector intercepts ...
2
votes
0answers
416 views
Create calibration plot with error bars for logistic regression model
I would like to create a calibration plot for a logistic regression model along with 95% confidence intervals for the mean predicted probability in each bin. The plot I'm after along with the code to ...
0
votes
0answers
27 views
Calibration plot with all points on one side of the line
I have a calibration plot from a logistic regression model in which all of the plotted points have an above-average predicted probability (i.e., each decile's mean predicted probability is above the ...
3
votes
0answers
61 views
inverse.predict chemcal package
I have noticed that the inverse.predict function (chemCal package) does not take into account all the degrees of freedom of the ...
2
votes
0answers
265 views
Calibration Curve v. Linear Dynamic Range [closed]
We have a piece of analytical chemistry equipment (a mass spectrometer for those who care), that has the capacity to generate a "Response value" (Area under the curve) that correlates to the amount of ...
0
votes
0answers
12 views
High-frequency recalibration of long-term prediction models
It seems that there is a practice (at lest in some sectors) to perform a regular "high frequency" (say, monthly) recalibration of parameters of models used for predicting over a long-term horizons (...
2
votes
0answers
247 views
Interpretation of this calibration curve on prediction models
I have several prediction models that I present in this calibration curve using R with the package "riskRegression".
In this study, the participants are community-dwelling older persons followed up ...
0
votes
0answers
103 views
Logistic regression on a perfectly classified dataset using platt calibration
I am reading up on how to calibrate a models outputs to have its probabilities make more sense. (link)
To quote excerpt from the paper (section 2.1):
If we use the same data set that was used to ...
1
vote
0answers
14 views
Detecting outlying instances from probability estimates
Say I have a binary classification model that is able to produce (posterior) probability estimates. I check whether these probabilities are calibrated and I evaluate using a metric that describes its ...
0
votes
0answers
18 views
Way to put variables on same scale across data sets
I have calculated four scores (say, A1/B1/C1/D1), where each score is made up granular variables in data set 1.
I have calculated the same four scores (say, A2/B2/C2/D2) in data set 2.
A1 and A2 are ...
8
votes
1answer
586 views
How to estimate a calibration curve with bootstrap (R)
Question: I have fitted a probabilistic model (bayesian network) for modeling a binary outcome variable. I would like to create a high-resolution calibration plot (e.g. spline) corrected for ...
0
votes
1answer
290 views
How to plot the calibration curve for an ordinal logistic regression model applied to a test sample?
I'm doing a validation study of an ordinal logistic regression model that was made with the lrm function of the rms package in R. How can I plot the calibration curve for the model when applied to new ...
1
vote
0answers
30 views
Goodness of Fit (weighted least squared) using variance instead of standard deviation
Is goodness of fit measured by weighted least square sometimes done by using the variance instead of the standard deviation in the denominator?
What I mean is the following. Lets consider the ...
2
votes
1answer
40 views
Correlation of model and experimental results
I am trying to find a scientifically accepted method to compare my model predictions with experimental values. Here, my model predicts the concentration of DNA as a function of time. DNA concentration ...
2
votes
0answers
109 views
Is there a maximum value for AUC for a given value of calibration?
Is there an upper boundary for the AUC of a risk model that is dependent on both the model calibration and the frequency of the outcome under examination?
Cook suggests that high values for ...
0
votes
1answer
29 views
Impact of the model error in numerical model prediction
I'm having trouble wrapping my head around characterizing model prediction uncertainties in the case of a calibrated numerical model. Let's assume we are trying to calibrate a set of input parameters ...
0
votes
1answer
69 views
Best way to check if two variables are correlated
(a) I have two variables that I am looking at: Visceral fat levels using MRI, and the visceral fat rating as shown by bioelectrical impedance (BIA) scales. The units for these two variables are ...
1
vote
0answers
188 views
R: Cox with time-dependent covariates Validation (discrimination/calibration) package [closed]
I am working on a dataset with multiple measures for each subject (id) and thus time-dependent covariates (measurement_value). I am interested in the "status" outcome. Some subjects do experience the ...
1
vote
1answer
124 views
How to match curves or measure the similarity between both?
Firstly, I'm not sure whether this is the correct site but I also assume the question is not too different:
Assume there are two curves: One is considered as a reference curve and the other is just ...
1
vote
0answers
18 views
Calibration of parametric multi-agent simulation to be consistent with emergent stylized facts
I've created a market simulation with a few different types of agents that trade with each other based on simple rules that sample from parametrized random distributions.
I can choose reasonable ...
2
votes
0answers
225 views
Implementing a very simple Bayesian Calibration
I want to understand Bayesian calibration.
I tried to implement a simple Bayesian calibration by constructing a set of truth data and then comparing my model to that data. My understanding of ...
1
vote
0answers
28 views
Methods on how to calibrate under/over-estimated probabilities
I have Dataset with some purchase probabilities on different products of different users.
After some comparison with observed purchase data for example by using conditional probability plots or ...
