Questions tagged [calibration]
Calibration can refer to predicting an explanatory variable from an observation of the dependent variable; to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc.
1
vote
0answers
26 views
What does the logistic calibration line represent on the calibration plot made by val.plot function (from rms package in R)?
I've been trying to make calibration plots/curves for a logistic regression model in R to mimic what I'm doing in a statistics course--but the class uses SAS with proc sgplot.
I think I've figured it ...
0
votes
1answer
17 views
Re-calibrating Intercept on logistic regression models for unbalance data
I have data-set that I’m modelling using logistic regression as land.cover~H1+H2+H3+H4+H6+H8+H14. My response and categorical variables are binary. However the number of 0 and 1 in my response ...
0
votes
0answers
26 views
Weights in “visited website” survey
I have a dataset with the browsing history of people in a country with age and gender, my goal is to give some information regarding some websites like :
the market shares between several website ...
0
votes
0answers
47 views
Survey weights: calibration with continuous variables
I'm using the R survey package and trying to run the calibrate function. My setup:
For each individual I have age, gender, ...
1
vote
0answers
28 views
Calibration curve for mixed model (logistic) [closed]
I did some searching using the CV search function and was unable to locate any information on R packages or general approach to creating a calibration curve with a bootstrapped curve overlay (similar ...
7
votes
2answers
177 views
Why is logistic regression well calibrated, and how to ruin its calibration?
In the scikit learn documents on probability calibration they compare logistic regression with other methods and remark that random forest is less well calibrated than logistic regression.
Why is ...
2
votes
0answers
39 views
Calibrating probabilities of a binary classifier when class prior is unknown
Is it possible to calibrate the probabilities of a binary classifier when the class priors are unknown?
In cases where the data is obtained with selection bias (i.e. more positives than negatives in ...
1
vote
0answers
43 views
How to validate(with sample-split data) and calibrate Cox model with time-dependent covaraites?
I am building 2 cox models:
Without time-dependent covariates
With time-dependent covariates.
1.The first model (without time-dependent variables) as specified below in R works fine and I have no ...
0
votes
2answers
114 views
Linear predictor from coefficients of Cox PH model
I need to calculate the linear predictor of a Cox PH model by hand.
I can get continuous and binary variables to match the output of predict.coxph (specifying 'lp') but I can't seem to figure out how ...
0
votes
0answers
60 views
Calculation of RMSEC (Calibration Error), Which data should be used?
I got a bit confused about RMSEC but I got how RMSECV and RMSEP are calculated and their meaning, though.
I am working based on 2 papers which summarize their results with RMSEC, RMSECV1, RMSECV2 and ...
0
votes
0answers
24 views
Sample Size Adjusted Metric for Calibration Curve
I'm creating a model that provides a predicted probability of an event happening. I am using calibration curves that plot predicted probability (x-axis) versus the actual event frequency (y-axis). I ...
3
votes
1answer
41 views
Measuring uncertainty of a fitted calibration curve
My question is: how can you compute the prediction interval of a calibration curve, just like you might for any other regression model.
A calibration curve maps estimated probabilities to empirical ...
2
votes
0answers
293 views
Probability calibration from LightGBM model with class imbalance
I've made a binary classification model using LightGBM. The dataset was fairly imbalnced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. ...
3
votes
0answers
131 views
Calibration of penalized (LASSO or ELasticNet) logistic regression models
I would be very grateful for any help me with the following general query regarding calibration of penalized models with a binary outcome.
I would like my prediction model to be calibrated (mean ...
1
vote
1answer
117 views
why is calibration of my logistic regression s shaped?
I am simulating data to compare real and predicted probabilities from logistic regression like this:
...
0
votes
1answer
20 views
Similarity between variables when using calibration curves
I'm reading through a text that is explaining calibration curves, and the following description is provided:
To be well-calibrated, the probabilities must effectively reflect the true likelihood of ...
1
vote
0answers
29 views
Boosted Regression Trees: Zero discrimination and calibration scores
I am using boosted regression trees using the gbm() and dismo() packages. When I run my models I get values of zero for discrimination and calibration in the $cv.statistics
What would be the reason ...
6
votes
1answer
427 views
Does Regularized Logistic Regression Produce Calibrated Results?
It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow ...
1
vote
1answer
382 views
Probability calibration metric for multiclass classifier
A machine learning classifier can be calibrated so that when the probability that datapoint i is of class A is 0.6, this is true 60% of the time.
In the binary class setting, this can be visualised ...
0
votes
1answer
211 views
Finding an optimal class probability threshold for SVM
I've got an imbalanced data set on which I'm training an SVM using cross validation. I'd like to find the optimal class probability threshold that maximizes the F measure.
I've tried doing this by ...
0
votes
1answer
37 views
Calibration of an individual-based model of an epidemic
I am currently developing an individual-based (or agent-based) mathematical model (IBM) of an epidemic. I want to calibrate the transmission parameters in my IBM to match empirical data (epidemic ...
1
vote
0answers
88 views
R caret calibration function
I'm studying about calibration techniques and I'd like to use the CARET calibration function to examine the quality of my classifiers' probabilities.
I read the relevant documentation, however I don'...
0
votes
0answers
17 views
Metric to determine distance of good pixels to a target value in an image with many defective pixels
An xray source emits radiation with strength being a function of some variables, one is kv. The xray strength is supposed to increase monotonically as kv increases.
Then an xray detector intercepts ...
2
votes
0answers
354 views
Create calibration plot with error bars for logistic regression model
I would like to create a calibration plot for a logistic regression model along with 95% confidence intervals for the mean predicted probability in each bin. The plot I'm after along with the code to ...
0
votes
0answers
25 views
Calibration plot with all points on one side of the line
I have a calibration plot from a logistic regression model in which all of the plotted points have an above-average predicted probability (i.e., each decile's mean predicted probability is above the ...
3
votes
0answers
41 views
inverse.predict chemcal package
I have noticed that the inverse.predict function (chemCal package) does not take into account all the degrees of freedom of the ...
2
votes
0answers
222 views
Calibration Curve v. Linear Dynamic Range [closed]
We have a piece of analytical chemistry equipment (a mass spectrometer for those who care), that has the capacity to generate a "Response value" (Area under the curve) that correlates to the amount of ...
0
votes
0answers
10 views
High-frequency recalibration of long-term prediction models
It seems that there is a practice (at lest in some sectors) to perform a regular "high frequency" (say, monthly) recalibration of parameters of models used for predicting over a long-term horizons (...
2
votes
0answers
212 views
Interpretation of this calibration curve on prediction models
I have several prediction models that I present in this calibration curve using R with the package "riskRegression".
In this study, the participants are community-dwelling older persons followed up ...
0
votes
0answers
96 views
Logistic regression on a perfectly classified dataset using platt calibration
I am reading up on how to calibrate a models outputs to have its probabilities make more sense. (link)
To quote excerpt from the paper (section 2.1):
If we use the same data set that was used to ...
1
vote
0answers
12 views
Detecting outlying instances from probability estimates
Say I have a binary classification model that is able to produce (posterior) probability estimates. I check whether these probabilities are calibrated and I evaluate using a metric that describes its ...
0
votes
0answers
16 views
Way to put variables on same scale across data sets
I have calculated four scores (say, A1/B1/C1/D1), where each score is made up granular variables in data set 1.
I have calculated the same four scores (say, A2/B2/C2/D2) in data set 2.
A1 and A2 are ...
7
votes
1answer
500 views
How to estimate a calibration curve with bootstrap (R)
Question: I have fitted a probabilistic model (bayesian network) for modeling a binary outcome variable. I would like to create a high-resolution calibration plot (e.g. spline) corrected for ...
0
votes
1answer
252 views
How to plot the calibration curve for an ordinal logistic regression model applied to a test sample?
I'm doing a validation study of an ordinal logistic regression model that was made with the lrm function of the rms package in R. How can I plot the calibration curve for the model when applied to new ...
1
vote
0answers
27 views
Goodness of Fit (weighted least squared) using variance instead of standard deviation
Is goodness of fit measured by weighted least square sometimes done by using the variance instead of the standard deviation in the denominator?
What I mean is the following. Lets consider the ...
2
votes
1answer
39 views
Correlation of model and experimental results
I am trying to find a scientifically accepted method to compare my model predictions with experimental values. Here, my model predicts the concentration of DNA as a function of time. DNA concentration ...
0
votes
0answers
80 views
Metrics for uncertainty estimates in steering angle prediction
I am working on steering angle prediction for self-driving cars. Assume the model has an output $\{\hat{y},\hat{\sigma}^2\}$ for the (continuous) steering angle $\hat{y}$ and variance $\hat{\sigma}^2$ ...
2
votes
0answers
87 views
Is there a maximum value for AUC for a given value of calibration?
Is there an upper boundary for the AUC of a risk model that is dependent on both the model calibration and the frequency of the outcome under examination?
Cook suggests that high values for ...
0
votes
1answer
29 views
Impact of the model error in numerical model prediction
I'm having trouble wrapping my head around characterizing model prediction uncertainties in the case of a calibrated numerical model. Let's assume we are trying to calibrate a set of input parameters ...
0
votes
1answer
56 views
Best way to check if two variables are correlated
(a) I have two variables that I am looking at: Visceral fat levels using MRI, and the visceral fat rating as shown by bioelectrical impedance (BIA) scales. The units for these two variables are ...
1
vote
0answers
150 views
R: Cox with time-dependent covariates Validation (discrimination/calibration) package [closed]
I am working on a dataset with multiple measures for each subject (id) and thus time-dependent covariates (measurement_value). I am interested in the "status" outcome. Some subjects do experience the ...
1
vote
1answer
112 views
How to match curves or measure the similarity between both?
Firstly, I'm not sure whether this is the correct site but I also assume the question is not too different:
Assume there are two curves: One is considered as a reference curve and the other is just ...
1
vote
0answers
18 views
Calibration of parametric multi-agent simulation to be consistent with emergent stylized facts
I've created a market simulation with a few different types of agents that trade with each other based on simple rules that sample from parametrized random distributions.
I can choose reasonable ...
2
votes
0answers
197 views
Implementing a very simple Bayesian Calibration
I want to understand Bayesian calibration.
I tried to implement a simple Bayesian calibration by constructing a set of truth data and then comparing my model to that data. My understanding of ...
1
vote
0answers
27 views
Methods on how to calibrate under/over-estimated probabilities
I have Dataset with some purchase probabilities on different products of different users.
After some comparison with observed purchase data for example by using conditional probability plots or ...
1
vote
1answer
382 views
Calibrating Probabilities worse than Original Model even though better performance on calibration curve? (in R / Caret)
I'm currently working on the exercises of the book 'Applied Predictive Modeling' by Kuhn and Johnson (using R and caret) and am stuck at the issue of 'Calibrating Probabilities'.
Exercise 12.3 shows ...
1
vote
0answers
27 views
How to combine purchase and click data togehter in sparse matrix
my problem is the following: I have purchase probability estimations of different products. The model behind don't take care of any inter-correlations through these products. So my task is to re-...
0
votes
1answer
45 views
How do I scale/standardize one set of data that is non-linear (due to temp variations in a device) to that of a data set that is linear?
I have data from two instruments over time. One of the instruments does not show a linear trend due to an anomaly but I want to standardize that data against that of the second instrument because I ...
1
vote
0answers
39 views
Regression vs Calibration [duplicate]
When reading on the wikipedia page of Calibration they said :
" A reverse process to regression, where instead of a future dependent variable being predicted from known explanatory variables, a ...
1
vote
1answer
549 views
Probability Calibration messes Reliability
I have about 1000 samples with 20 features and I'm using Random Forest to predict a binary class.
I'm trying to apply the probability calibration process as described on scikit using ...
