Questions tagged [calibration]
Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.
198
questions
1
vote
0answers
8 views
How many points needed to compute the Homography? [migrated]
I'm working on a project where i'm using planar homography.
As seen in the above image, every point gives two equations and since the homography matrix has 8 degree of freedom, 4 points are enough to ...
1
vote
0answers
20 views
Should Scikit-Learn CalibratedClassifierCV isotonic mode use bucketed rates instead of the actual targets?
This is less a question about sklearn's implementation, and more theoretical. I find it weird that we'd do isotonic regression against target values in {0, 1} because that could result in very jagged ...
0
votes
0answers
9 views
Re-calibrate interaction matrix in population sample
Let's say I have a population {$(G, p)$} where $G$ is a group within the population, and $p$ is their proportion distributed as such:
...
0
votes
0answers
12 views
AUC for crossvalidation
I have a medical research scenario where I am trying to predict disease progression. I need to produce a model to integrate into clinical decision support (and evaluate further). In addition to ...
0
votes
0answers
13 views
Proper way to incorporated CalibratedClassifierCV in cross-validation in Scikit
I'm creating some classifiers for a binary classification problem. I want to find out three things:
Which algorithm I should use.
Which set of hyperparameters I
should use.
If I should calibrate the ...
0
votes
0answers
19 views
How can I draw calibration curve for Cox model in external validation set
Dear all: I had builded a nomogram from Cox proportional hazards model, now I want to do calibration for this nomogram in external validation set， how can I achieve it in r? Thanks!
1
vote
1answer
31 views
Can a binary output model with auc 0.5 be perfectly calibrated?
I am reading up on model calibration and I stumbled upon this article. To quote:
We can have a perfectly accurate model that is not calibrated at all and, on the other hand, a model that is no better ...
0
votes
0answers
19 views
Calibration measure for classification with linear slope
I would like to know if there is a measure for calibration, in binary classification case, that is global, and not only a visual one, like reliability/calibration plot/curve.
In particular, in another ...
0
votes
0answers
26 views
Binary classification on imbalanced data - odd calibration curve
I have a dataset with 1MM records, around 40 features and 2 classes. The incidence of class '1' is only 1.8%. I am in need of (a)good ROC AUC (in the range .70-.85) and (b)good probability predictions ...
0
votes
0answers
6 views
comparing calibration across two models
I have two models that predict a binary outcome. The range of model A is [0, 0.2] while the range of model B is [0, 1] with very sparse high probability predictions. Using the typical decile binning ...
0
votes
0answers
14 views
Is there an error in this paper loss?
I'm reading Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers and in section 3.1 they describe their entropy margin loss. The goal of this loss is to make the ...
2
votes
1answer
69 views
Measuring predictive uncertainty with Negative Log Likelihood (NLL)?
I see that in many papers about prediction uncertainty and calibration of neural networks, methods are compared in terms of the negative log-likelihood. What does it represent in this context? And why ...
5
votes
1answer
112 views
probability calibration and Brier score
Assume that I have a binary classification problem. The outcome from classification I am mostly interested in is the well-calibrated probabilities.
The first way to check this is the calibration plot (...
1
vote
0answers
33 views
LASSO Logistic Regression & Platt Scaling
I recently built a cross-validated (10 folds) LASSO logistic regression model to make binary predictions within STATA. Interestingly, the model calibration was initially poor:
Realizing that the ...
1
vote
1answer
84 views
calibration of classifier scores: isotonic regression
I am investigating the isotonic regression approach to calibrate the scores from a classifier.
If I understand correctly, we do the following. First, we get the calibration plot (or reliability curve),...
0
votes
0answers
31 views
calibrate using “rms” with polspline error
I am working on a cox model using the "rms" package by Prof.@Frank Harrell' and I would like to measure on calibration and discrimination. However, I am facing an error polspline during ...
0
votes
1answer
47 views
Logistic regression risk prediction model - poor calibration but good discrimination
I am trying to create risk prediction model in R. I am new to logistic regression risk prediction analysis. I obtained reliability curve using ...
0
votes
0answers
15 views
Can we do inverse prediction for data having multi-class response variable after fitting PLS-DA?
I know how to do inverse prediction (predicting one of the input variables when we know what is the output we want) for the case of regression. I know we can do the same for binary classification ...
0
votes
0answers
26 views
Why average probability estimates when applying Platt scaling with cross validation?
On the subject of doing probabalistic classification and calibration with cross validation, the sklearn docs for Probability Calibration state:
...
1
vote
0answers
28 views
What smoothing parameter makes sense for a LOESS calibration curve?
I am creating a calibration curve to asses the fit of a logistic regression.
Does it make more sense to use the local or global optimum smoothing parameter for the LOESS line?
The orange line uses ...
0
votes
0answers
9 views
What does the detection limit indicate?
Say I have a data set of 6 samples that give me the concentration per mL for each sample.
Data:
5 g/ml
7 g/ml
3 g/ml
4 g/ml
5 g/ml
6 g/ml
Here the standard deviation is 1.414. The critical value for ...
4
votes
0answers
44 views
Calibrating devices
I need to calibrate physical devices to align their measurements to a common nominal base. Figure below depicts curves of measurements to be aligned.
As it can be seen from the picture the difference ...
0
votes
1answer
35 views
why is calibration not needed for a logistic regression with only categorical features?
Frank Harrell states in one of the answers, "...Note that if the model contains only categorical variables and interactions among the variables are not needed, the model must fit the data and no ...
0
votes
1answer
99 views
How to get a confidence interval around the output of logistic regression?
I'm doing logistic regression with two classes (A and B), and I'd like to be able to describe the outputs of the model in terms of (calibrated) probabilities that each sample is in class A or B. If ...
3
votes
1answer
28 views
Confidence Interval around a predictor
I have a logistic regression as follows: $\log \frac{p}{1-p} = \beta_0 + \beta_1x$.
I'm looking for a confidence interval around a value of $x$, which would correspond to a specific value of $p$. ...
2
votes
1answer
45 views
Ranking most probable labels from multilabel classifier
I have been working on a multilabel classification problem. I want to classify whether each of 25 labels is present on a given sample. The labels are not mutually exclusive. Ultimately, I would like ...
0
votes
0answers
16 views
Should I calculate the classification performances (AUROC etc) before or after the neural network calibration?
Should I calculate the classification performances (AUROC, AUPR etc) before or after the neural network calibration (using for ex. isotonic regression)?
0
votes
0answers
19 views
Discrimination vs calibration
So far I have been using logistic regression for binary classification problems usually for unbalanced classes - and would resort to the standard F1 score, AUROC, and Gini to compare and contrast the ...
2
votes
0answers
26 views
Calibration in linear, logistic and Poisson regression
I read the following in Google's Rules of ML:
In linear, logistic, or Poisson regression, there are subsets of the data where the average predicted expectation equals the average label (calibrated). ...
0
votes
0answers
21 views
What's moment calibration?
I read the following in Google's Rules of ML:
In linear, logistic, or Poisson regression, there are subsets of the data where the average predicted expectation equals the average label (1-moment ...
0
votes
0answers
41 views
Calibration of RF classifier: with sample_weight vs. without sample_weight
I am working with Random Forest binary classifier and use isotonic regression (using CalibratedClassifierCV from sklearn) for probability calibration.
The question: assuming that RF classifier is ...
1
vote
0answers
22 views
1
vote
0answers
68 views
Understanding probability calibration with isotonic regression in sklearn
After reading sklearn manual it was not very obvious for me to understand how Isotonic regression works in the case of probability calibration (using CalibratedClassifierCV).
I briefly read sklearn's ...
0
votes
0answers
10 views
Why should I use a calibration curve? If the curve for my model doesn't look right, is my model useless?
I've been scouring the web for more information on calibration curves. Scikit-learn has probably the best documentation I've found thus far. Here's their description:
When performing classification ...
1
vote
1answer
70 views
Confidence interval limits out of range
I have constructed a calibration curve to measure a pollutant concentration in water. My calibration standards ranged from 2 to 20 mg/L. I used a blank (0 mg/L).
I have obtained a predicted ...
0
votes
0answers
19 views
Choice of metric for calibration
I am having a hard time to understand what calibration metric to pick for a given task. I am reading a lot to compare the different existing calibration measures but still can't find anything that ...
0
votes
0answers
23 views
Calibration slope- what are the units for interpretation of (mis)calibration?
I am working to validate a time-to-event model with the linear predictor and baseline hazard provided. I wish to report the calibration slope. However, the units for interpretation are unclear. I am ...
0
votes
0answers
8 views
Calibration and consistency for predictions over time
It's common to plot the predicted probability of some event/events versus the actual fraction of such events that happened to obtain a calibration curve showing whether the predictions were under or ...
2
votes
0answers
101 views
Platt Scaling vs Isotonic Regression
I am learning classifier probability calibrations and have calibrated an eleastic net model using both Platt scaling and isotonic regression. As you can see in the attached image Platt scaling (on the ...
1
vote
0answers
120 views
Calibration plot for survival analysis (Cox regression) using R [closed]
I have been working on developing a Cox regression model and was trying to create a calibration plot after binning the predicted risks into deciles of risk. I have the following code:
...
2
votes
1answer
433 views
How is slope calculated in a calibration plot?
I am using logistic regression with white cell count and temperature as predictors and hospital admission>3 days as the outcome of interest. I'm using the rms package in R to assess calibration (curve ...
2
votes
1answer
111 views
Inverse Regression vs Reverse Regression
I'm aware there's a great number of questions which deal with the mathematical difference between the two, but I'm still confused as to best practice.
Basically I'm looking at a situation where we ...
1
vote
0answers
17 views
Mutliple Regression Calibration Curve
Background:
I am using an instrument that measures two physical properties, X1~Temperture and X2~ Velocity. When gathering the data to make the curve a set of predetermined concentrations are chosen ...
0
votes
0answers
20 views
Any paper proposing calibration to increase Kolmogorov Smirnov distance in a binary classification?
Consider a traditional classification settings, when we already came up with a model parameter $\theta$ ( by Logistic Regression as example). We also have the scores $f(x, \theta |y=1) \in R $ for ...
0
votes
0answers
11 views
Help with weighting sample according to population
I am a beginner with basic knowledge of statistics - just learning. I have a doubt regarding weighting survey sample distribution to population distribution.
I have to create a weighting variable that ...
1
vote
0answers
49 views
Person's parameter (ability) calibration using R's package ltm
I have a user response ($0/1$) matrix user_item_matrix, which has $1000$ rows (users) and $30$ columns (items/questions). There is a lot of missing data (more than $...
0
votes
0answers
16 views
Expected output distribution of a calibrated model on controls-only test set
I have a question related to ML model calibration. I train a well calibrated binary classification model with the two classes being controls and cases. The model outputs a number between 0 and 1 that ...
0
votes
0answers
24 views
What methods work for assigning population into groups while matching average parameters
I have $N=1000$ population size which is distributed into $K=10$ groups (A, B, C..) with a different number of people in each. Every person belongs to one and only one group.
We know weight has 4 ...
1
vote
0answers
29 views
Error propagation through a calibration curve
Say I have a linear fit given y = ax + b. I'm given Δa and Δb as 95% confidence intervals.
I now have several measurements y1, y2, y3, ... etc, from which of course I can gather a mean, standard ...
2
votes
0answers
89 views
Calibration-in-the-small and logistic regression
It is well agreed (for example this discussion ) that Logistic Regression guarantees that model will produce well calibrated in the large (mean) predictions.
Does Logistic Regression guarantees ...
