20

There are two things mentioned in the CalibratedClassifierCV docs that hint towards the ways it can be used: base_estimator: If cv=prefit, the classifier must have been fit already on data. cv: If “prefit” is passed, it is assumed that base_estimator has been fitted already and all data is used for calibration. I may obviously be interpreting this ...


15

I am interested in this question as well and wanted to add some experiments to better understand CalibratedClassifierCV (CCCV). As has already been said, there are two ways to use it. #Method 1, train classifier within CCCV model = CalibratedClassifierCV(my_clf) model.fit(X_train_val, y_train_val) #Method 2, train classifier and then use CCCV on ...


12

Cox models do not predict outcomes! "Best" methods depend on whether you obtain a risk score (as with Framingham) or absolute risk (as with Gail Breast Cancer Risk). You need to tell us exactly what you're fitting With absolute risk prediction, you can split groups according to their risk deciles and calculate proportions of observed vs. expected outcome ...


11

The term ``calibration'', as applied to survey weights, appears to have been coined by Deville and Sarndal (1992). They put an umbrella on a bunch of different procedures that used the known population totals: $$ \sum_{i \in \mathcal{U}} Y_i = T_i $$ where $Y_i$ is a vector of characteristics known for every unit in the population $\mathcal{U}$. For ...


8

Setup Recall that an SVM can be viewed as a weight vector $w$ and an intercept $b$, and that the output function for a test input $x$ is is $\langle w, x \rangle + b$. To get a binary prediction, we take $f(x) = \mathrm{sign}(\langle w, x \rangle + b)$. (I'm going to use some primal notations here, but use $\langle \cdot, \cdot \rangle$ to denote that ...


7

A straightforward way to calibrate Cox survival models is to use the calibrate function provided by the rms package in R, as in the page that you linked. This package provides a cph method for Cox models that is designed to work with the calibration and validation methods that it provides for several types of regression models. Quoting from the manual page: ...


7

After discussing with prof Frank Harrell by email, I devised the following procedure for estimating the optimism-corrected calibration curve, partially based on his Tutorial in Biostatistics (STATISTICS IN MEDICINE, VOL. 15,361-387 (1996)): fit a risk prediction model on all data fit a flexible model (gam with spline and logit link) to the model's predicted ...


7

The answer depends entirely on the amount of penalization used. If too little, the model will be seen to be overfitted when evaluated on an independent sample. If too much penality, it will be found to be underfitted. The goal is to solve for the penalty that gets it "just right". Two ways to do this are cross-validation (e.g., 100 repeats of 10-fold ...


6

Let $x$ be the original time series and $x_m$ be the result of smoothing with a simple moving average with some window width. Let $f(x, \alpha)$ be a function that returns a smoothed version of $x$ using smoothing parameter $\alpha$. Define a loss function $L$ that measures the dissimilarity between the windowed moving average and the exponential moving ...


6

I'm not sure "the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated" is correct: gradient boosting tends to push probability toward 0 and 1. Furthermore, you're applying weights, which should also skew your probabilities. Because gradient boosting pushes probabilities outward rather than inward, using ...


5

It looks to me like it might potentially make sense to account for the way the error changes with the second (calibrated) val as well as with speed. If you're only interested in the relationship with speed, it looks like linear regression would be a good first suggestion. First, let's look at your data; this is every variable against every other variable, ...


5

Logistic regression is a classification method that basically learns a probability function $\pi_\theta(x)$ over the input space by fitting the parameters $\theta$. If the predicted probabilities are learned with the appropriate loss function, than logistic regression has the potential to learn an unbiased estimation of the binary event probabilities, ...


5

The short answer is that it only makes sense to calculate the Brier score for the conditional probabilities, $\hat y = P(y=1|X)$, where $y$ is the outcome, $\hat y$ is your prediction, and $X$ are your predictors. In other words, $\hat y$ is the probability that $y=1$, conditional on this particular value of the predictors, $X$. The Brier score in this case ...


4

Another option is isotonic regression. It is similar to whuber's answer except the bins are generated dynamically instead of by splitting in halves, with a requirement that outputs are strictly increasing. This primary usage of isotonic regression is to recalibrate your probabilities if they are shown to be poorly calibrated, but it can also be used for ...


4

You should be aware that with calibration your independent variable is the concentration of your standards and the dependent variable is the measured quantity. Look at package chemCal for some nice functions. concentration <- c(27.2,32.4,16.5,11.6,11.9,9.87,46.0,73.6,75.4,73.1,59.5,49.0, 79.0,81.6,66.7,26.7) absorbance <- c(0.764, ...


4

Based on your comments, what you want to do is a calibration, which you also want to validate: you have reference measurements of a temperature (thermometer A), and measurements of instrument B which is not a thermometer yet, as you do not get response of the physical quantity temperatures but of a physical quantity like e.g. electrons/s. Camera readout is ...


4

You are doing a (chemical) calibration, and the search phrase you are looking for is method validation in analytical chemistry. There actually exist norms how to validate methods in analytical chemistry, and certain measures of performance like limit of detection (LOD), limit of quantitation (LOQ, probably more relevant for you), recovery rate, etc. If you ...


4

I believe the bias in RFs is "a feature, not a bug." Decision trees have a tendency to overfit the data, so the bias in random forests counter-acts the overfit somewhat, making a random forest model more robust than a straight-forward decision tree otherwise would have been. If you want to further attack overfit, you may be interested in using regularized ...


4

yes, the R* (root...) versions are the square root of the corresponding MSEs (mean squared errors) They differ in the type of cases that are used to measure them: RMSEC: calibration error, i.e. the residuals of the calibration data. (R)MSEC measures goodness of fit between your data and the calibration model. Depending on the type of data, model and ...


4

If you want to measure overall performance, I recommend approaches like nested cross-validation instead of comparing the output of models trained on different folds. The latter isn't the way to go. I don't recommend Platt scaling in general. If you really need probabilities, don't use an SVM. You are almost always far better off with (kernel) logistic ...


4

Any statistical method that uses binning has ultimately been deemed obsolete. Continuous calibration curve estimation has been commonplace since the mid 1990s. Commonly used methods are loess (with outlier detection turned off), linear logistic calibration, and spline logistic calibration. I go into this in detail in my Regression Modeling Strategies book ...


4

As you know the Brier score measures calibration and is the mean square error, $\bar B = n^{-1} \sum (\hat y_i - y_i)^2$, between the predictions, $\hat y,$ and the responses, $y$. Since the Brier score is a mean, comparing two Brier scores is basically a comparison of means and you can go as fancy with it as you like. I'll suggest two things and point to a ...


4

Although this question and its first answer seems to be focused on theoretical issues of logistic regression model calibration, the issue of: How could one ruin the calibration of a logistic regression...? deserves some attention with respect to real-world applications, for future readers of this page. We shouldn't forget that the logistic regression ...


3

The behavior of estimators calibrated to estimated control totals has been studied already, see Dever and Valliant 2010 (which is the most technical part of Jill Dever's dissertation). They derive the bias explicitly, discuss the (somewhat weird) conditions under which the bias is small, and provide additional variance terms necessary to account for ...


3

Later in that section, there is an example where the posterior mean using the inferential prior is larger than the posterior mean using the true prior, and this is said to be an example of positive miscalibration. Therefore I think the intended definition of miscalibration is: $$ \text{miscalibration} = \text{(posterior mean using inferential prior)} - \text{...


3

Sorry I didn't see this question when you first posed it. Any method that requires binning of continuous variables is to be avoided. The often-used method of stratifying predicted values and then computing ordinary Kaplan-Meier estimates was always problematic - you could see this by how noisy the results appeared. I never use calibration methods based ...


3

the point is to pick a value of the calibration parameter that gives optimum model correlation In that case, the question is about model optimization, which is related to, but usually not considered the same, as data dredging. Fortunately, there are "recipes" that you can follow to avoid the problems associated with the dredging. The big problem with ...


3

Your assumption that the measures will only differ by a multiplicative constant strikes me as certainly false. The fact that this would not work for converting from Fahrenheit to Celsius demonstrates that. (A.k.a. #3) You will need to assess more than one part. You will not have enough degrees of freedom to determine the conversion between the two ...


3

If you make the less restrictive assumption that the two measurements are related by some linear equation, then: For question 1, you can assess the assumption using linear regression. If it is valid, the intercept should be 0 (or very close to 0, if there is measurement error). For question 2, the coefficient will tell you the constant to use I am not ...


Only top voted, non community-wiki answers of a minimum length are eligible