Effects of ignoring the sampling strategy

Question

Does the use of calibration/post-stratification imply that the sampling strategy/stratification/individual selection probabilities can be ignored altogether?

In a survey reweighting exercise, I have population totals for a complete cross-classification of four attributes: country, age, sex, and work status. EDIT: The sampling scheme is stratified by country (about the same number of respondents per country), and within each country stratified by age and sex (very roughly uniform). The work status is not part of the sampling stratification.

My task is to produce generic weights that can be used for identifying interactions between variables (stratification-response and response-response). Estimation of population means or totals might be another application. (At least that's what I understand. To be honest, I don't really know the application -- how much does it matter?)

Question: How will the calibrated weights be affected if information on the sampling strategy is added to the calibration procedure, e.g. in the form of prior weights? Is it a problem when for some combinations of categories there are only very few respondents?

Question related to the practical exercise: https://stats.stackexchange.com/q/29132/6432

Answer 1

There may be more advanced weighting schemes that I'm not familiar with, but this is based on simple weighting schemes I've used with small numbers of groups

How will the calibrated weights be affected if information on the sampling strategy is added to the calibration procedure, e.g. in the form of prior weights: I don't know what this means. There are only two groups of information you need to weight the data. You have your "golden" population sizes for each group (a group being, for example, employed males in the US age 18-24, estimated from an external data source assumed to be true) and your sampled sizes for each group. By multiplying the population percentage by the sample size for each group, you will get to the intended weighted sample size. Divide the weighted sample size by the actual sample size to get at the weight to apply for that group. Easy peasy.

(I can insert an example table, but I'm not sure how to put a table in my response like I've seen before. Can somebody help in a comment?)

Is it a problem when for some combinations of categories there are only very few respondents?

Yes. There are two problems when the weight for a group is significantly greater than 1 (1 being the case where no weighting is needed).

First, it may reflect a problem in the sampling. Weighting doesn't correct for significantly different people who were excluded from the sampling frame. It assumes that the people you missed were substantially the same as the people you got. If you conduct a landline telephone survey and weight by age to correct for the under-sampling of 18-24 year-olds, that doesn't really work, because the few young people who have landlines will be significantly different on almost any metric from the majority that don't.

Second, it can overstate the reliability of a finding. If the weighted sample size for a group is 100, it can look like a certain margin of error. But, the actual margin of error is based on the unweighted sample size, which may be 50 if that group had a weight of 2. For that reason, you should report the unweighted sample size for each metric. Most survey reporting software makes it easy to report weighted metrics and unweighted sample sizes simultaneously.