Permissible to use survey data in generating sampling weights?

Question

Summary: Can I use non-population information (e.g. representative proportions from another survey) when calculating sampling weights? If so how might one account for the sampling error?

I'm calculating standard errors for a set of mean estimates (cohorts are age, gender and health state).

I can calculate sampling weights using my data and population age, gender information from the census. There is no census information on health state.

I have non-census health status information from a 'large' survey* (from the same population) which also has age-gender information. Presently I assume that the 'large' survey* point estimates for relative proportions of health states (for a given age gender cohort) are sufficiently precise to stand in for the population proportions which I do not observe. I use these alongside census age/gender information to derive population cohort counts and calculate sampling weights as usual. Then I use Stata's svy package for se's via Taylor linearization.

I'm uncomfortable with the stand-in of the the 'large' survey* proportions in place of true population proportions as the imprecision of these is not accounted for in my standard error estimation.

I think the influence of these is small, but how acceptable would you say the above approach is? Are there better ways of going about it?

When I say sampling weights I use the following simple formula: (Npopulation of a given age, gender, health cohort)/Npopulation)/(nsample of a given age, gender, health cohort/nsample)

The larger survey is the National Survey of Mental Health and Welbeing (Australia), which has a stratified, multistage, design. The smaller survey is a simple random sample.

Answer 1

You have four reasons that the standard errors you compute naively are wrong.

1. Weights
2. Stratification
3. Clustering
4. Calibration

The first three is the standard triumvirate of complex surveys; you can read about them here, there and everywhere. In a nutshell, unequal weights generally increase the standard errors; stratification generally decreases them; and clustering generally increases them (and often is the greatest source of precision loss).

When you calibrate your survey to another survey or a census, the standard errors have to be calculated differently. Think about this: if you adjusted the weights to match the population proportions of males and females, then you have zero sampling error when you "estimate" that proportion from your own data. So your estimation routine, whatever it is, must split out a zero for the standard error. If it does not, this is not the right estimation method.

The general theory of variance estimation with calibrated survey data is due to Deville and Sarndal (1992). At a very basic (and not particularly accurate) level, instead of taking the difference $y_{hi}-\bar y_h$ in the standard survey variance calculations, you need to first "filter" your data by regressing the outcome on the calibration variables and taking residuals, so the variance involves the sums of $e_{hi}-\bar e_h$ instead.

Calibration to the data from other surveys that are themselves subject to sampling error is more difficult yet. Dever and Valliant (2010) had a paper on this, demonstrating who to properly account for an inflation in standard errors. This is generally difficult; what I would do (and I think I have done this in the past on a couple of obscure problems of this nature) would be to generate independent complex survey bootstrap samples for both your own survey and the calibration target survey, estimate the totals from the latter, and use these perturbed totals as calibration targets generating the bootstrap weights for my own survey. Dever and Valliant talk about jackknife, but they are old school :). I like the bootstrap better, as it requires fewer assumptions.

Answer 2

I would argue that your estimates from the large survey are sufficiently precise. A 99% confidence interval around your estimates would include a margin of error of $2.576\times\sqrt{\frac{0.5(1-0.5)}{8820}}\approx1.37\%$ if you're being as conservative as possible and assume $\hat{p}=0.5$. In many cases, this will actually be much smaller. It depends on the level of precision you need and the potential drawbacks if your estimates are inaccurate, but I believe that the weights you use here will be close enough to the true values that your end product will not be significantly affected.

If precision is worrisome to you, I might suggest using a stratified random sample as opposed to a purely simple random sample. This will likely help you to gain the precision that you might lose by estimating these weights from the larger survey and not from the population as a whole.