Skip to main content

Questions tagged [calibration]

Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.

Filter by
Sorted by
Tagged with
3 votes
1 answer
110 views

Why would `pROC::roc` calculate $\max\{AUC, 1 - AUC\}$ by default?

There is some interesting behavior in the pROC::roc function in R. ...
Dave's user avatar
  • 64.7k
0 votes
1 answer
26 views

Rank the sensitivity among multiple calibration curves

Let's say that, with a measurement device, we have a linear relationship between an output measurement I (in mV) and the concentration ...
Basj's user avatar
  • 632
6 votes
1 answer
41 views

What is happening behind the scenes when we use CalibratedClassifierCV without prefit?

From what I understood by reading sklearn Probability Calibration, when we run CalibratedClassifierCV we will fit "a regressor (called a calibrator) that maps the output of the classifier (as ...
andy mot's user avatar
0 votes
0 answers
17 views

How to test for statistical significance on a multi-country dataset?

I am attempting to test for statistical significance between treatment and control groups, with data from 7 countries, using a Welch 2-sample t-test [t.test() in R]. 6 out of 7 countries have ...
Corey Chin's user avatar
1 vote
0 answers
89 views

Calibrating CatBoostClassifier produces worse results

I'm performing multiclass probability prediction using CatBoostClassifier on a dataset with ~4000 rows, 13 features, 4 target classes. Dataset has outliers, but it is balanced. For this task I'm using ...
primadonna's user avatar
0 votes
0 answers
31 views

How can I assess internal validation (discrimination, calibration) of a Fine-and-Gray competing risk model, fitted using a MFP-algoritm in Stata?

I'm a post-doc at Karolinska Institute, and I'm working on developing a Fine-and-Gray competing risk model to predict endometrial cancer recurrence/progression with death as a competing event. I have ...
Rasmus Green's user avatar
7 votes
1 answer
122 views

What distribution assumptions do Gupta, Podkopaev & Ramdas (NeuroIPS 2020) think could be made?

A 2020 NeuroIPS paper by Gupta, Podkopaev & Ramdas addresses the calibration of outputs to binary “classification” models, admitting that the raw scores, despite perhaps being on $\left[0, 1\right]...
Dave's user avatar
  • 64.7k
1 vote
1 answer
26 views

Y axis of calibration plot: incidence per X vs percentage at risk

I am considering showing how mis-calibrated a cox proportional hazard model is by plotting the 10th percentiles of risk on the x axis vs the incidence per 100,000. For each bin in x I could plot data ...
brucezepplin's user avatar
1 vote
1 answer
72 views

How to evaluate multi-class classifier on probability prediction task?

I have a balanced dataset where each object (song) has one of the four target class labels (mood of a song). Example: ID feature1 feture2 feature3 target_class 0 0.5 0.11 125 upbeat 1 0.23 0.75 136 ...
primadonna's user avatar
0 votes
1 answer
102 views

Calibration in the large for continuous outcomes

I'm a bit confused around calibration in the large. I usually see it discussed in the context of binary outcomes, but am I correct in thinking it can also be valuable as a part of external validation ...
JeffR's user avatar
  • 1
3 votes
1 answer
63 views

How can model overconfidence coincide with accurate classifications?

Guo et al (ICML 2017) state the following. During training, after the model is able to correctly classify (almost) all training samples, NLL can be further minimized by increasing the confidence of ...
Dave's user avatar
  • 64.7k
2 votes
0 answers
44 views

How should I combine “typical” and “yesterday” self-reports?

I have inherited a long-running survey with two measures of individual behavior. Edits: clarifying that this is not about drinking behavior; it’s not, and I only used that to try and illustrate. It ...
dholstius's user avatar
  • 101
0 votes
1 answer
39 views

Hosmer-Lemeshow Calibration error when predicted probabilities are clustered

I have extracted predicted probabilities (logistic model) from a graph according to the nine classes of a certain variable (I don't own the model). I need to compare the predicted probabilities, that ...
vixxovs's user avatar
  • 45
1 vote
0 answers
80 views

LightGBM Regressor miscalibratred/underestimating on high fitted values and overestimating on low fitted values

I'm training a pretty standard LightGBM regressor and noticing a strange pattern with the residuals (see images below--I'm bunching the predicted values and taking the observed average for the group). ...
dfried's user avatar
  • 201
0 votes
0 answers
6 views

Methods for delaying the "break" in non-linear least squares optimisation when the step size gets too small?

I am using the Levenberg-Marquardt method for calibration purposes. Typically, the RMSE of my calibration looks like: I want to break the algorithm when the algorithm step-updates start to slow down, ...
THATS MY QUANT MY QUANTITATIVE's user avatar
4 votes
0 answers
91 views

Can the calibration-discrimination decomposition of Brier score be viewed as the bias-variance decomposition of mean squared error?

The mean squared error has a famous decomposition into bias and variance. $$ \text{MSE} = \text{bias}^2 + \text{var} $$ Brier score is also a mean squared error calculation, and Brier score has a ...
Dave's user avatar
  • 64.7k
1 vote
0 answers
33 views

How to quantify the quality of a graphed calibration curve?

In his Is Medicine Mesmerized by Machine Learning? blog article, Frank Harrell shows a calibration curve (below) and states that it is quite poor. I follow the logic: the claimed probability of $0.20$...
Dave's user avatar
  • 64.7k
1 vote
0 answers
43 views

Temperature scaling a bayesian neural network?

I am trying to calibrate a Bayesian neural network. I have already approximated the posterior density for its weights. In order to make predictions the Bayesian way, I am taking samples from the ...
Randomdude's user avatar
0 votes
0 answers
26 views

Calibrating a non-homogeneous Poisson process to my data [duplicate]

My question: Let's say I have some data on the cumulative number of infections per day since the start of a pandemic at $t=0$. Since clearly the infection rate changes over time, I want to calibrate a ...
Michaël's user avatar
1 vote
1 answer
21 views

How to diagonalise when there is less parameters to estimate than data in the Levenberg-Marquardt algorithm

I am trying to calibrate a Heston Model with 100 call options using this paper https://arxiv.org/pdf/1511.08718.pdf. In algorithm 4.1 on page 18, they define the dampening factor as: $$\mu_0 = \omega \...
THATS MY QUANT MY QUANTITATIVE's user avatar
0 votes
0 answers
26 views

Comparing proxy metrics to human evaluations

I have two proxy metrics, and I'd like to see which of them correlates more strongly with human ratings. I have ~30 questions, and for each question 3 humans independently give a score on a 1-10 scale....
augray's user avatar
  • 101
2 votes
1 answer
142 views

XGBoost Calibration for weighted loss function

I am currently using XGBoost (in R) to perform multiclass classification. I am using merror=eval_metric and my objective is <...
HeyCool08's user avatar
0 votes
0 answers
37 views

Update/ recalibrate XG Boost, Random Forest, GLM models for external validation

I have created XG Boost, Random Forest and GLM models for classification of a binary outcome and now I want to externally validate the models on a different population of over 5000 subjects. I have ...
mmo's user avatar
  • 11
2 votes
1 answer
92 views

Do uncalibrated "probability" predictions satisfy Kolmogorov's axioms?

Let's say we have some binary variable of interest and fit a model to predict the probability of the two classes, say a logistic regression or a "classification" neural network. This model ...
Dave's user avatar
  • 64.7k
1 vote
0 answers
123 views

Is perfect isotonic probability calibration realistic?

I work with a labelled tabular dataset of about 1 million observations, with the target being binary. The dataset is heavily imbalanced - about 0.5% positive class. I have trained a gradient boosting ...
StrLdn's user avatar
  • 11
6 votes
0 answers
119 views

Understanding a calibration plot for lightGBM binary classifier

I wanted to assess the performance of my lightGBM classifier using a calibration plot. If I understood correctly, a calibration plot visualizes the alignment between the predicted probabilities by the ...
Programming Noob's user avatar
1 vote
0 answers
54 views

Assessing uncertainty calibration in regression using the CDF

I have a labelled data set with $n$ data points $(x_i, y_i)$ with $x_i \in \mathbb{R}^k$ and $y_i \in \mathbb{R}$ and I trained a model $f: \mathbb{R}^k \to \mathbb{R} \times \mathbb{R}^+$ on some of ...
PascalIv's user avatar
  • 829
1 vote
0 answers
33 views

How to get standard error from constrained optimization problem in R?

Can I get standard error from a constrained optimization problem in R? I have calculated transition probabilities. Now I am trying to calibrate it. Using these transition probabilities I have ...
Md. Zubab Ibne Moid's user avatar
0 votes
1 answer
138 views

Model performance with multiply imputed data

I would like to know how to do calibration plot with Hosmer-lemeshow test and AUC for ROC curve after multiple imputation in R. I build one prediction model and tried to do model performance but ...
Haruka Hayashi's user avatar
1 vote
0 answers
90 views

Model calibration in overfitted models

Why in Shrinkage, due to an overfitted prediction model, do we tend to overestimate risk for "high risk" subjects and to underestimate risk for "low risk" subjects ? Intuitively I ...
vixxovs's user avatar
  • 45
1 vote
1 answer
169 views

Optimizing a threshold value on a dependent metric using a classifier trained to optimize a threshold-independent metric

Is it a reasonable approach to train a probabilities classifier by optimizing a threshold-independent metric such as AUC, and then using the trained classifier to calibrate the decision threshold ...
Amit S's user avatar
  • 57
0 votes
0 answers
58 views

Calibration plot without binning predictions

Similar to ths question I would like to know how to create a calibration curve without binning my predictions. What makes my situation different, is that I'm using icenReg for my interval-censored ...
Wojty's user avatar
  • 75
0 votes
1 answer
85 views

How does someone achieve a desired confidence / accuracy when measuring using uncalibrated instrument?

I have an instrument that measures a value. It is only possible to measure the value once i.e. the experiment can't be repeated (think recording a car's speed as it drives past). The instrument is not ...
Chuck's user avatar
  • 81
0 votes
0 answers
21 views

Question on Calibrated Structured Prediction (Kuleshov, 2015)

I have several questions regarding equation (2) from the paper: Calibrated Structured Prediction (Kuleshov, 2015). I believe most of my questions stem from not understanding the quantity: $T(x)=\...
confused_grad_student's user avatar
1 vote
0 answers
166 views

Softmax Response vs MC Dropout for Uncertainty Estimation [closed]

Some papers I see take the uncertainty estimation of a prediction as simply its softmax/sigmoid output, whereas some papers will use techniques such as MC Dropout and calculate the variance across the ...
user8903610's user avatar
1 vote
0 answers
671 views

Creating and interpreting calibration plots for several models with a binary outcome

I have made several models (RF, XGB and GLM) to predict a binary outcome and they all achieved an AUC of approximately 0.8 and Brier scores 0.1-0.15. Test set is fairly small (n= 350), cases with ...
mmo's user avatar
  • 11
5 votes
1 answer
3k views

Create calibration plot in R with vectors of predicted and observed values

I am currently working on a project regarding the external validation of a logistic regression model for binary classification. I would like to create a calibration plot and compute the calibration-in-...
Yan Linxuan's user avatar
4 votes
3 answers
523 views

Low p-value, but many trials

I am not well-versed in statistics, so please bear with me. I will try to explain things to the best of my knowledge. I have some hardware that produces a signal, this signal has an offset and I want ...
user17004502's user avatar
0 votes
0 answers
75 views

Model for probability threshold selection, how to make differentiable loss?

I have an object detection model that outputs bounding boxes and probabilities associated with them. The probability expresses the chance that the object found is a tumor. I have noticed that the ...
Slajni's user avatar
  • 45
2 votes
2 answers
542 views

Should sum of sample weights always be equal to the target population

I have a simple question regarding sample weighting: From what I understand, sample weights are used to reduce or eliminate potential biases arising from the differences between the selected sample ...
Guoqiang Zhang's user avatar
3 votes
1 answer
902 views

Why do we need separate data for probability calibration?

Why do we need separate data for probability calibration? Scikit learn documentation says: The samples that are used to fit the calibrator should not be the same samples used to fit the classifier, as ...
Glue's user avatar
  • 485
1 vote
0 answers
37 views

Neural Networks Miscalibration Measure

I have read these two papers related to the neural network miscalibration problem: "On Calibration of Modern Neural Networks" and "Multivariate Confidence Calibration for Object ...
alimagadovk's user avatar
1 vote
1 answer
49 views

Does probability calibration descrease model prediction variance?

Does probability calibration decrease model prediction variance? Example: Let's say we have a classifier that is a mail spam detector. It outputs a score between 0-1 to quantify how likely a given ...
Glue's user avatar
  • 485
2 votes
0 answers
72 views

Bayesian calibration of computer simulations - Likelihood function calculation

I am starting to study Bayesian calibrations of computer models. I am not a statistician and just starting to learn so bear with me if I do not use the correct terminology. The general approach is ...
Rojj's user avatar
  • 121
2 votes
0 answers
394 views

Does scikit-learn support plotting calibration curve for multiclass classifier?

I have trained a multi class classifier and calibrated the classified using scikit-learn library's CalibratedClassifierCV class. To check how well the probabilities are calibrated, I tried plotting ...
yodasoda18's user avatar
3 votes
1 answer
3k views

Probability Calibration for Highly Imbalanced Binary Classification

I am working on a binary classification problem on a highly imbalanced dataset (1:100) where model probabilities are important for the use case and need to be well calibrated to best represent true ...
Jake Niederer's user avatar
1 vote
1 answer
68 views

Is the true conditional $P(Y \mid X)$ calibrated?

I am trying to get an intuitive understanding of the concept of calibration. Definitions first. Consider a data distribution $P(X, Y)$ over binary labels $Y$, and a probabilistic classifier which ...
usual me's user avatar
  • 1,247
1 vote
0 answers
261 views

How do you do decision threshold tuning when doing k-fold cross validation?

I'm training a binary classifier for disease detection. Because of my small amount of data (~1000 datapoints, 10% positive, 90% negative), I've realized that doing an 80-20 train-test split produces ...
jimbo's user avatar
  • 11
1 vote
1 answer
691 views

Correct way to interpret sklearn's calibration error and generate a numeric calibration loss/score

I have been using sklearn's CalibrationDisplay and think it is pretty cool. One thing I am wondering, though, is how I could potentially take that curve and make it an interpretable score. For example,...
bismo's user avatar
  • 121
3 votes
1 answer
89 views

Probability Calibration, what causes simple linear models like logistic regression to be over-confident and diverge from the true class probabilities?

I've recently studied probability calibration and have investigated many examples and revisited old models of mine to find that they are all poorly calibrated. The idea of stacking also has been cast ...
Oliver's user avatar
  • 310

1
2 3 4 5
7