Estimating bias in surveys

Say a company runs a survey across random N cities independently in some country estimating the fraction of males and females on each city.

E.g.:

• Males = $X_1$% and Females = $(100 - X_1)$% for city 1
• Males = $X_2$% and Females = $(100 - X_2)$% for city 2
• etc.

Now, say that we have a independent and unbiased estimate of the fraction of males and females in the entire country, i.e. Males = $Y_T$%, Females = $(100-Y_T)$%.

Using all of this data, how can we get a potentially better estimate of the fraction of males and females in each city?

What would be a frequentist or Bayesian way of solving this problem?

• Information about the population as a whole is of little help unless you can specify the relation between these cities and the population. If you could say that the cities combined ARE the whole population, then there would be a way to adjust the estimates via some version of post-stratification. However, without the clear relation between the cities and the population, you are risking that your correction will only make matters worse. Apr 29 '15 at 15:09

Here's a solution using survey sampling theory ("frequentist" solution).

Let's assume that the estimate of males and females in the entire country is a good one (its variance is sufficiently low). Then the estimate in each city can be improved using calibration techniques (such as post-stratification or Deville and Särndal's). The estimate of number of males and females in the country is then called a calibration margin.

Calibration will give you new weights, which define the calibrated estimator, which is :

• also unbiased, as long as you have a sufficient number of units in your survey
• more precise (precision increases with correlation between calibration margin and survey variables, which is likely high in your case)

If the quality of the estimation in the entire country is very poor (variance is much higher than for the estimation in the two cities), then it can't be used as a calibration margin.

• What about shrinkage and pooling? And what if we assume that bias is constant across all estimates? Are you sure there is no benefit to pooling
– Josh
Apr 29 '15 at 17:05
• @Josh. Shrinkage and pooling is the direction I was thinking of but I think you need the sample size and population size for all surveys to get an idea of the standard deviations of your estimates, $X_i$ and $Y_T$. Apr 30 '15 at 1:45
• Added an edit to answer you question regarding pooling. May 1 '15 at 7:47
• Thanks @AntoineR I added a note to the question that can perhaps simplify the problem (let's assume that the country-wide estimate is unbiased).
– Josh
May 12 '15 at 19:33
• It does simplify the answer : see my edit. May 14 '15 at 12:38