What does it mean to calibrate survey weights?
Also, what are other definitions of calibration in statistics? I have heard it used in several contexts, particularly risk prediction (referring to whether the total number of predicted events in a cohort is statistically consistent with the observed number events).
Is there a grand, unifying notion of calibration in statistics?
Our wiki on calibration scratches the surface, or perhaps begs the question.
The term ``calibration'', as applied to survey weights, appears to have been coined by Deville and Sarndal (1992). They put an umbrella on a bunch of different procedures that used the known population totals:
$$ \sum_{i \in \mathcal{U}} Y_i = T_i $$
where $Y_i$ is a vector of characteristics known for every unit in the population $\mathcal{U}$. For general human populations, these would be census data on demographic characteristics, like age, gender, race/ethnicity, education, geography (regions, states, provinces), and may be income. For establishment populations, these variables typically have to do with establishment size and income. For list samples -- whatever it is that you have attached to your sample.
Deville and Sarndal (1992) discussed how to go from design weights (inverse probabilities of selection), $d_i, i\in \mathcal{S}$ where $\mathcal{S}$ is the sample drawn from $\mathcal{U}$, to calibrated weights $w_i$ such that
$$ \sum_{i \in \mathcal{S}} w_i y_i = T_i $$
i.e., the sample agrees with the population on these variables. They did this by optimizing a distance function
$$ F(w_i,d_i) \ge 0 \to \min, \quad F(r_i,r_i) = 0, \mbox{ subject to } \sum_{i \in \mathcal{S}} w_i y_i = T_i $$
Typically, as is often the case in statistics, bringing in additional information improves variances asymptotically, although may throw things off and introduce weird small sample biases. Deville and Sarndal (1992) quantified these asymptotic efficiency gains, which was their central contribution to the literature.
In regards of using auxiliary data, survey statistics stands as a pretty unique branch. Bayesian folks utilize auxiliary data in their priors. The i.i.d. frequentists / likelihoodists usually don't have much of a way to incorporate auxiliary information, it seems, as all information must be contained in the likelihood. There is, however, a branch of empirical likelihood estimation where auxiliary information is being used to generate and/or aggregate estimating equations; in fact the empirical likelihood objective functions is one of the objective function cases considered by Deville and Sarndal (1992). (Econometricians should sniff, quite properly, and point out that they have known the ways to calibrate statistical models via generalized method of moments for more than 30 years by now, since Hansen (1982). A quadratic loss is another naturally interesting case in Deville and Sarndal (1992); while it is the easiest to compute, it can give rise to negative weights that are usually considered weird.)
Another use of the term ``calibration'' in statistics that I heard of is the reverse regression, in which you have inaccurate measurements of the variable of interest and want to recover the value of the predictor (the running example I was given by my marathon runner of a stats professor was measuring the distance of the course by biking it and counting the revolutions of the bike wheels, vs. more accurate GPS measurements -- that was in the late 1990s era before smarthphones and handheld GPS devices.) You calibrate your bike on an established 1km course, and then try to bike around to get 42 that much.
There may be yet other uses. I am not sure dumping them all in one entry is particularly wise though. You indicated factor analysis as one potential user of that term, but I am not particularly well aware of how it is used there.
say you run a survey and get 1,000 responses. maybe you conducted your survey via cell phone and older people don't have cell phones at the same rate as younger people. so 5% of your survey respondents (N=50) were senior citizens but maybe according to the united states census bureau, 15% of americans are actually senior citizens. and let's say respondent age is going to matter for your survey analysis, for the stuff you ultimately publish. in order for your 1,000 responses to properly generalize to a more realistic population, you need to give your senior citizens weights of 3x (the weighted N needs to be 150) and scale everyone else's weights down a bit (the nonelderly N=950 should have a weighted N of 850). calibration, raking, post-stratification are techniques to bring your survey data set closer to some official total. this is a simple case, but most survey weights are re-computed based on more than just one factor (in this case, age)
There are other uses of the term "calibration." For instance in this CV thread, Frank Harrell discusses it in the context of determining model fit:
The key thing to check first is the model's calibration, either using the bootstrap to correct for overfitting or using a huge independent sample not used for model development or fitting.
Understanding how good a prediction is, in logistic regression
Typically, in predictive modeling, calibration of a model refers to evaluating the goodness-of-fit (or model accuracy) on the training data, whereas predictive error is evaluated on the test data.
I'd like to share with you an intuitive example to explain the following statement:
we desire that the estimated class probabilities are reflective of the true underlying probability of the sample. That is, the predicted class probability (or probability-like value) needs to be well-calibrated. To be well-calibrated, the probabilities must effectively reflect the true likelihood of the event of interest.
Source: Applied Predictive Modeling on page 249
For a particular case it is hard to illustrate the probability, but when it comes to large size of cases the effect of calibration appears.
For instance, the airline company use some algorithms, for instance the logistic regression(why?), to predict whether the passenger will show up on that day. They actually don't care about if the particular person will turn up or not, and what they concern is how many shows would be in total. What they do can be just adding all the probabilities of the predictions. If the sum is bellow the number of the seats they can overbook flights.
Some classifiers are not well-calibrated for instance SVM. That's the score we get while predicting is not the true probability.
Calibration means that you set desired parameters with known good values which would result in expected outcomes. Calibrated values are well established values. The confidence of its values, which as depicted in the calibration curve σ, shows the confidence in the estimated values as deviation from the mean μ. Here, 68% of values are within one standard deviation σ away from the mean; while 95% of the values lie within two sigmas.
