Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [calibration]

Calibration can refer to predicting an explanatory variable from an observation of the dependent variable; to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc.

-1
votes
0answers
12 views

Calibration vs. Estimation in RBC Models [on hold]

What are the pro(s) and con(s) of estimation and calibration in RBC models in macro. Can you compare it?
2
votes
0answers
34 views

Calibrating probabilities of a binary classifier when class prior is unknown

Is it possible to calibrate the probabilities of a binary classifier when the class priors are unknown? In cases where the data is obtained with selection bias (i.e. more positives than negatives in ...
1
vote
0answers
18 views

How to validate(with sample-split data) and calibrate Cox model with time-dependent covaraites?

I am building 2 cox models: Without time-dependent covariates With time-dependent covariates. 1.The first model (without time-dependent variables) as specified below in R works fine and I have no ...
0
votes
2answers
54 views

Linear predictor from coefficients of Cox PH model

I need to calculate the linear predictor of a Cox PH model by hand. I can get continuous and binary variables to match the output of predict.coxph (specifying 'lp') but I can't seem to figure out how ...
0
votes
0answers
29 views

Calculation of RMSEC (Calibration Error), Which data should be used?

I got a bit confused about RMSEC. I got how RMSECV and RMSEP are calculated and their meaning, though. I am working based on 2 papers which summarize their results with RMSEC, RMSECV1, RMSECV2 and ...
0
votes
0answers
20 views

Sample Size Adjusted Metric for Calibration Curve

I'm creating a model that provides a predicted probability of an event happening. I am using calibration curves that plot predicted probability (x-axis) versus the actual event frequency (y-axis). I ...
3
votes
0answers
31 views

Measuring uncertainty of a fitted calibration curve

My question is: how can you compute the prediction interval of a calibration curve, just like you might for any other regression model. A calibration curve maps estimated probabilities to empirical ...
1
vote
0answers
127 views

Probability calibration from LightGBM model with class imbalance

I've made a binary classification model using LightGBM. The dataset was fairly imbalnced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. ...
0
votes
0answers
13 views

model calibration in complex data

I am working with a complex dataset (national inpatient sample) which has weight, cluster and stratum variables. My aim is to look for predictors of pediatric post-operative respiratory failure. I ...
3
votes
0answers
71 views

Calibration of penalized (LASSO or ELasticNet) logistic regression models

I would be very grateful for any help me with the following general query regarding calibration of penalized models with a binary outcome. I would like my prediction model to be calibrated (mean ...
1
vote
1answer
66 views

why is calibration of my logistic regression s shaped?

I am simulating data to compare real and predicted probabilities from logistic regression like this: ...
0
votes
1answer
19 views

Similarity between variables when using calibration curves

I'm reading through a text that is explaining calibration curves, and the following description is provided: To be well-calibrated, the probabilities must effectively reflect the true likelihood of ...
1
vote
0answers
21 views

Boosted Regression Trees: Zero discrimination and calibration scores

I am using boosted regression trees using the gbm() and dismo() packages. When I run my models I get values of zero for discrimination and calibration in the $cv.statistics What would be the reason ...
6
votes
1answer
342 views

Does Regularized Logistic Regression Produce Calibrated Results?

It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow ...
1
vote
1answer
241 views

Probability calibration metric for multiclass classifier

A machine learning classifier can be calibrated so that when the probability that datapoint i is of class A is 0.6, this is true 60% of the time. In the binary class setting, this can be visualised ...
0
votes
1answer
153 views

Finding an optimal class probability threshold for SVM

I've got an imbalanced data set on which I'm training an SVM using cross validation. I'd like to find the optimal class probability threshold that maximizes the F measure. I've tried doing this by ...
0
votes
1answer
34 views

Calibration of an individual-based model of an epidemic

I am currently developing an individual-based (or agent-based) mathematical model (IBM) of an epidemic. I want to calibrate the transmission parameters in my IBM to match empirical data (epidemic ...
1
vote
0answers
53 views

R caret calibration function

I'm studying about calibration techniques and I'd like to use the CARET calibration function to examine the quality of my classifiers' probabilities. I read the relevant documentation, however I don'...
0
votes
0answers
15 views

Metric to determine distance of good pixels to a target value in an image with many defective pixels

An xray source emits radiation with strength being a function of some variables, one is kv. The xray strength is supposed to increase monotonically as kv increases. Then an xray detector intercepts ...
2
votes
0answers
261 views

Create calibration plot with error bars for logistic regression model

I would like to create a calibration plot for a logistic regression model along with 95% confidence intervals for the mean predicted probability in each bin. The plot I'm after along with the code to ...
0
votes
0answers
22 views

Calibration plot with all points on one side of the line

I have a calibration plot from a logistic regression model in which all of the plotted points have an above-average predicted probability (i.e., each decile's mean predicted probability is above the ...
2
votes
0answers
12 views

inverse.predict chemcal package

I have noticed that the inverse.predict function (chemCal package) does not take into account all the degrees of freedom of the ...
2
votes
0answers
138 views

Calibration Curve v. Linear Dynamic Range [closed]

We have a piece of analytical chemistry equipment (a mass spectrometer for those who care), that has the capacity to generate a "Response value" (Area under the curve) that correlates to the amount of ...
0
votes
0answers
10 views

High-frequency recalibration of long-term prediction models

It seems that there is a practice (at lest in some sectors) to perform a regular "high frequency" (say, monthly) recalibration of parameters of models used for predicting over a long-term horizons (...
2
votes
0answers
177 views

Interpretation of this calibration curve on prediction models

I have several prediction models that I present in this calibration curve using R with the package "riskRegression". In this study, the participants are community-dwelling older persons followed up ...
0
votes
0answers
84 views

Logistic regression on a perfectly classified dataset using platt calibration

I am reading up on how to calibrate a models outputs to have its probabilities make more sense. (link) To quote excerpt from the paper (section 2.1): If we use the same data set that was used to ...
1
vote
0answers
12 views

Detecting outlying instances from probability estimates

Say I have a binary classification model that is able to produce (posterior) probability estimates. I check whether these probabilities are calibrated and I evaluate using a metric that describes its ...
0
votes
0answers
16 views

Way to put variables on same scale across data sets

I have calculated four scores (say, A1/B1/C1/D1), where each score is made up granular variables in data set 1. I have calculated the same four scores (say, A2/B2/C2/D2) in data set 2. A1 and A2 are ...
7
votes
1answer
417 views

How to estimate a calibration curve with bootstrap (R)

Question: I have fitted a probabilistic model (bayesian network) for modeling a binary outcome variable. I would like to create a high-resolution calibration plot (e.g. spline) corrected for ...
0
votes
1answer
221 views

How to plot the calibration curve for an ordinal logistic regression model applied to a test sample?

I'm doing a validation study of an ordinal logistic regression model that was made with the lrm function of the rms package in R. How can I plot the calibration curve for the model when applied to new ...
1
vote
0answers
25 views

Goodness of Fit (weighted least squared) using variance instead of standard deviation

Is goodness of fit measured by weighted least square sometimes done by using the variance instead of the standard deviation in the denominator? What I mean is the following. Lets consider the ...
2
votes
1answer
36 views

Correlation of model and experimental results

I am trying to find a scientifically accepted method to compare my model predictions with experimental values. Here, my model predicts the concentration of DNA as a function of time. DNA concentration ...
0
votes
0answers
71 views

Metrics for uncertainty estimates in steering angle prediction

I am working on steering angle prediction for self-driving cars. Assume the model has an output $\{\hat{y},\hat{\sigma}^2\}$ for the (continuous) steering angle $\hat{y}$ and variance $\hat{\sigma}^2$ ...
1
vote
0answers
70 views

Is there a maximum value for AUC for a given value of calibration?

Is there an upper boundary for the AUC of a risk model that is dependent on both the model calibration and the frequency of the outcome under examination? Cook suggests that high values for ...
0
votes
1answer
28 views

Impact of the model error in numerical model prediction

I'm having trouble wrapping my head around characterizing model prediction uncertainties in the case of a calibrated numerical model. Let's assume we are trying to calibrate a set of input parameters ...
0
votes
1answer
55 views

Best way to check if two variables are correlated

(a) I have two variables that I am looking at: Visceral fat levels using MRI, and the visceral fat rating as shown by bioelectrical impedance (BIA) scales. The units for these two variables are ...
1
vote
0answers
124 views

R: Cox with time-dependent covariates Validation (discrimination/calibration) package [closed]

I am working on a dataset with multiple measures for each subject (id) and thus time-dependent covariates (measurement_value). I am interested in the "status" outcome. Some subjects do experience the ...
0
votes
0answers
27 views

Analyzing data from non random sample, specific example

Scenario: Imagine that a software company provides inventory management software for hardware stores across the US. Their market penetration is approximately 30% of all stores, and the true number of ...
1
vote
1answer
89 views

How to match curves or measure the similarity between both?

Firstly, I'm not sure whether this is the correct site but I also assume the question is not too different: Assume there are two curves: One is considered as a reference curve and the other is just ...
1
vote
0answers
18 views

Calibration of parametric multi-agent simulation to be consistent with emergent stylized facts

I've created a market simulation with a few different types of agents that trade with each other based on simple rules that sample from parametrized random distributions. I can choose reasonable ...
2
votes
0answers
173 views

Implementing a very simple Bayesian Calibration

I want to understand Bayesian calibration. I tried to implement a simple Bayesian calibration by constructing a set of truth data and then comparing my model to that data. My understanding of ...
1
vote
0answers
25 views

Methods on how to calibrate under/over-estimated probabilities

I have Dataset with some purchase probabilities on different products of different users. After some comparison with observed purchase data for example by using conditional probability plots or ...
1
vote
1answer
331 views

Calibrating Probabilities worse than Original Model even though better performance on calibration curve? (in R / Caret)

I'm currently working on the exercises of the book 'Applied Predictive Modeling' by Kuhn and Johnson (using R and caret) and am stuck at the issue of 'Calibrating Probabilities'. Exercise 12.3 shows ...
1
vote
0answers
27 views

How to combine purchase and click data togehter in sparse matrix

my problem is the following: I have purchase probability estimations of different products. The model behind don't take care of any inter-correlations through these products. So my task is to re-...
0
votes
1answer
35 views

How do I scale/standardize one set of data that is non-linear (due to temp variations in a device) to that of a data set that is linear?

I have data from two instruments over time. One of the instruments does not show a linear trend due to an anomaly but I want to standardize that data against that of the second instrument because I ...
1
vote
0answers
27 views

Regression vs Calibration [duplicate]

When reading on the wikipedia page of Calibration they said : " A reverse process to regression, where instead of a future dependent variable being predicted from known explanatory variables, a ...
1
vote
1answer
486 views

Probability Calibration messes Reliability

I have about 1000 samples with 20 features and I'm using Random Forest to predict a binary class. I'm trying to apply the probability calibration process as described on scikit using ...
5
votes
0answers
498 views

Signatures of underfitting and overfitting in logistic regression calibration curves

My confusion stems from reading the following paper http://www.bmj.com/content/351/bmj.h3868 It states in its abstract (and they later show an empirical study that conforms to the claim) - "...
0
votes
0answers
87 views

maximum likelihood estimation for Generalized normal error distribution

In short, I want to understand why using Laplacian distribution to model errors performs better than using Gaussian distribution to model errors. In detail, I am using Generalized normal distribution ...
1
vote
0answers
35 views

Only ROC reported for predictive model

I have just finished reading a medical paper that claims to have developed a predictive model but reported ROC alone without a calibration curve. ROC seems to be useful for cases of binary ...