Questions tagged [calibration]
Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.
216
questions
1
vote
0answers
13 views
Why Negative Log Likelihood (NLL) is a measure of model's calibaration?
In the context of (multiclass) classification, I've read papers which imply that NLL is minimized iff the model is well-calibrated (outputing true probability for each class and not just confidence), ...
0
votes
0answers
37 views
+50
Correct approach to probability classification of a binary classifier
After having read a few articles and papers on probability calibration, I still don't have a clear understanding of what could be the best way to do model calibration in my case.
I am using LightGBM (...
0
votes
0answers
8 views
Best Way to Report Calibration
I'm wondering what is the most accepted way to report metrics for model calibration. I have seen R2, slope and the brier score being used.
In addition, are there any packages in Python that can be ...
0
votes
1answer
8 views
Proper way to perform cross-validatation for selecting best parameters to build a calibrated model and assessing the error of the model?
I want to find the best possible:
Post process to apply on my data (for example, whether or not perform PCA or scaling, to remove some features...). Some of this options have parameters to tune (like ...
0
votes
1answer
23 views
XGBoost, Imbalanced Data and CalibratedClassifierCV
I am currently working with a slightly imbalanced dataset (9% positive outcome) and am using XGBoost to train a predictive model.
...
0
votes
1answer
17 views
Intepretation of a probabilistic classifier's result
I understand that a probabilistic classifier predicts the probability distribution $P(C|X)$. However I do not understand what $P(C|X)$ means.
Is it "$P(C=c|x)$ the probability of belonging to ...
1
vote
0answers
15 views
Is AUC or calibration error comparable among different subsets with different number of samples if I calculate them on subsets of test set?
When we fit a machine learning model, we will use metrics like AUC or calibration error to check the model probability ranking estimate or probability estimate on the test dataset. I tried to ...
0
votes
0answers
15 views
Model calibration
I have a logistic model estimate and I want to use it to score a new data. I know every variable in my new data is calculated correctly as the model build data except one variable. How can I calibrate ...
0
votes
0answers
25 views
Brier score loss and Log loss cut-offs [duplicate]
I read in multiple posts about the benefits of optimising the Brier score loss or the Log loss during cross validation since it helps to find the most robust model. Nevertheless, is there a cut-off ...
1
vote
1answer
40 views
CalibratedClassifier and RandomSearchCV
I was wondering what the right steps would be to perform both hyperparameter optimisation and obtain a calibrated model.
I thought the following could be the right way (70% train split, 10% validation ...
0
votes
1answer
40 views
Are there constraints for the variance of predicted probability on calibrated models?
I'm sorry if the title is too vague. I'm not really sure of what I ask, this is a somewhat speculative question...
The setting is that I'm using XGBoost in a binary classification problem (40% ...
0
votes
0answers
19 views
EM Algorithm For Distributions That Share Parameters
I've implemented an EM algorithm for a gaussian mixture where the mean is defined by an Ornstein–Uhlenbeck process. The pdf I'm trying to maximize is:
$$f(r(s)|r(t)) = (1-q)\cdot e^{-(r(s) - r(t) - k(\...
5
votes
1answer
223 views
When a classifier predicting probability should be calibrated?
At scikit-learn website they have a very nice picture showing the need to calibrate [some] classifiers to correct bias in predicted probabilities:
And they have a very nice explanation of why one ...
0
votes
0answers
47 views
How is confidence defined in Expected Calibration Error?
I'm building a Bayesian Neural Network, and am trying to understand how to calibrate the uncertainty estimates. From a paper by Seedat and Kanan (https://chriskanan.com/wp-content/uploads/seedat2019....
0
votes
0answers
8 views
How to validate or calibrate confidence intervals for binary outcomes?
One can calibrate binary classification predictions by quantizing the predictions into buckets and comparing against the target mean. But how can one go about calibrating CI's or estimates of ...
0
votes
0answers
6 views
if i want well calibrated probabilities but have class imbalance what metric?
i am having some issues on trying to get a correct metric for an imbalanced problem. it is a credit risk problem where i am trying to predict default of a company so i care about probability output.
i ...
0
votes
0answers
16 views
What is the right way to interpret over and under forecast in calibration curves?
I'm studying calibration curves and I'm stuck with a question regarding interpretation.
What (I believe) I have understood so far can be summarized as follows: some classification algorithms produce ...
2
votes
1answer
75 views
Observed probabilities in logistic regression?
Calibration is important performance metric in predictive modeling. My question is about calibration plot in logistic regression. Observed values, say, in linear model are those which are actually ...
3
votes
1answer
107 views
Is it a good idea to continue training a model after the train/validation accuracy has stopped improving?
The following animated diagram shows the training statistics of a Deep Neural Network classifier at the end of each epoch:
The diagrams on the left show the accuracy (upper) and loss (lower) values ...
0
votes
1answer
23 views
Why can we train calibrators using the original labels?
When using calibrators (e.g. Platt Scaling, Isotonic Regression) to get better calibrated probabilities from our classifiers, we are effectively finding a mapping from the output scores of the ...
2
votes
0answers
92 views
Why is sklearn's CalibratedClassifierCV not labeled as an ensemble method? [closed]
I always wondered how CalibratedClassifierCV was supposed to achieve probability calibration without a dedicated calibration set (which is appealing since no data is lost for training the classifier). ...
3
votes
1answer
119 views
Should Scikit-Learn CalibratedClassifierCV isotonic mode use bucketed rates instead of the actual targets?
This is less a question about sklearn's implementation, and more theoretical. I find it weird that we'd do isotonic regression against target values in {0, 1} because that could result in very jagged ...
0
votes
0answers
9 views
Re-calibrate interaction matrix in population sample
Let's say I have a population {$(G, p)$} where $G$ is a group within the population, and $p$ is their proportion distributed as such:
...
0
votes
0answers
21 views
AUC for crossvalidation
I have a medical research scenario where I am trying to predict disease progression. I need to produce a model to integrate into clinical decision support (and evaluate further). In addition to ...
1
vote
1answer
196 views
Proper way to incorporated CalibratedClassifierCV in cross-validation in Scikit
I'm creating some classifiers for a binary classification problem. I want to find out three things:
Which algorithm I should use.
Which set of hyperparameters I
should use.
If I should calibrate the ...
0
votes
0answers
54 views
How can I draw calibration curve for Cox model in external validation set
Dear all: I had builded a nomogram from Cox proportional hazards model, now I want to do calibration for this nomogram in external validation set， how can I achieve it in r? Thanks!
1
vote
1answer
43 views
Can a binary output model with auc 0.5 be perfectly calibrated?
I am reading up on model calibration and I stumbled upon this article. To quote:
We can have a perfectly accurate model that is not calibrated at all and, on the other hand, a model that is no better ...
0
votes
0answers
41 views
Calibration measure for classification with linear slope
I would like to know if there is a measure for calibration, in binary classification case, that is global, and not only a visual one, like reliability/calibration plot/curve.
In particular, in another ...
0
votes
0answers
226 views
Binary classification on imbalanced data - odd calibration curve
I have a dataset with 1MM records, around 40 features and 2 classes. The incidence of class '1' is only 1.8%. I am in need of (a)good ROC AUC (in the range .70-.85) and (b)good probability predictions ...
0
votes
0answers
10 views
comparing calibration across two models
I have two models that predict a binary outcome. The range of model A is [0, 0.2] while the range of model B is [0, 1] with very sparse high probability predictions. Using the typical decile binning ...
0
votes
0answers
14 views
Is there an error in this paper loss?
I'm reading Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers and in section 3.1 they describe their entropy margin loss. The goal of this loss is to make the ...
3
votes
1answer
354 views
Measuring predictive uncertainty with Negative Log Likelihood (NLL)?
I see that in many papers about prediction uncertainty and calibration of neural networks, methods are compared in terms of the negative log-likelihood. What does it represent in this context? And why ...
5
votes
1answer
217 views
probability calibration and Brier score
Assume that I have a binary classification problem. The outcome from classification I am mostly interested in is the well-calibrated probabilities.
The first way to check this is the calibration plot (...
1
vote
0answers
74 views
LASSO Logistic Regression & Platt Scaling
I recently built a cross-validated (10 folds) LASSO logistic regression model to make binary predictions within STATA. Interestingly, the model calibration was initially poor:
Realizing that the ...
1
vote
1answer
166 views
calibration of classifier scores: isotonic regression
I am investigating the isotonic regression approach to calibrate the scores from a classifier.
If I understand correctly, we do the following. First, we get the calibration plot (or reliability curve),...
0
votes
0answers
50 views
calibrate using “rms” with polspline error
I am working on a cox model using the "rms" package by Prof.@Frank Harrell' and I would like to measure on calibration and discrimination. However, I am facing an error polspline during ...
0
votes
1answer
103 views
Logistic regression risk prediction model - poor calibration but good discrimination
I am trying to create risk prediction model in R. I am new to logistic regression risk prediction analysis. I obtained reliability curve using ...
0
votes
1answer
58 views
Measuring the confidence of a softmax classification outcome
Suppose I have a softmax distribution produced by a classifier. There are four labels, and so the sum of the softmax probabilities over the four labels will be 1.0.
I am looking for a measurement of ...
0
votes
0answers
19 views
Can we do inverse prediction for data having multi-class response variable after fitting PLS-DA?
I know how to do inverse prediction (predicting one of the input variables when we know what is the output we want) for the case of regression. I know we can do the same for binary classification ...
0
votes
0answers
43 views
Why average probability estimates when applying Platt scaling with cross validation?
On the subject of doing probabalistic classification and calibration with cross validation, the sklearn docs for Probability Calibration state:
...
1
vote
0answers
50 views
What smoothing parameter makes sense for a LOESS calibration curve?
I am creating a calibration curve to asses the fit of a logistic regression.
Does it make more sense to use the local or global optimum smoothing parameter for the LOESS line?
The orange line uses ...
0
votes
0answers
9 views
What does the detection limit indicate?
Say I have a data set of 6 samples that give me the concentration per mL for each sample.
Data:
5 g/ml
7 g/ml
3 g/ml
4 g/ml
5 g/ml
6 g/ml
Here the standard deviation is 1.414. The critical value for ...
4
votes
0answers
44 views
Calibrating devices
I need to calibrate physical devices to align their measurements to a common nominal base. Figure below depicts curves of measurements to be aligned.
As it can be seen from the picture the difference ...
0
votes
1answer
41 views
why is calibration not needed for a logistic regression with only categorical features?
Frank Harrell states in one of the answers, "...Note that if the model contains only categorical variables and interactions among the variables are not needed, the model must fit the data and no ...
0
votes
1answer
998 views
How to get a confidence interval around the output of logistic regression?
I'm doing logistic regression with two classes (A and B), and I'd like to be able to describe the outputs of the model in terms of (calibrated) probabilities that each sample is in class A or B. If ...
3
votes
1answer
29 views
Confidence Interval around a predictor
I have a logistic regression as follows: $\log \frac{p}{1-p} = \beta_0 + \beta_1x$.
I'm looking for a confidence interval around a value of $x$, which would correspond to a specific value of $p$. ...
2
votes
1answer
67 views
Ranking most probable labels from multilabel classifier
I have been working on a multilabel classification problem. I want to classify whether each of 25 labels is present on a given sample. The labels are not mutually exclusive. Ultimately, I would like ...
0
votes
0answers
18 views
Should I calculate the classification performances (AUROC etc) before or after the neural network calibration?
Should I calculate the classification performances (AUROC, AUPR etc) before or after the neural network calibration (using for ex. isotonic regression)?
0
votes
0answers
23 views
Discrimination vs calibration
So far I have been using logistic regression for binary classification problems usually for unbalanced classes - and would resort to the standard F1 score, AUROC, and Gini to compare and contrast the ...
2
votes
0answers
38 views
Calibration in linear, logistic and Poisson regression
I read the following in Google's Rules of ML:
In linear, logistic, or Poisson regression, there are subsets of the data where the average predicted expectation equals the average label (calibrated). ...
