Estimating the true distributions from a sample of distributions

Question

I am having a hard time formulating the following problem. Consider a company that runs a survey across several cities in the US to estimate the percentage of right-handed people and left-handed people each city (we can ignore that ambidextrous people exist for this problem).

We are told that the the counts are biased (presumably with the same bias across all cities), but we don't know how exactly or how much.

The input are the counts for a long list of cities:

city, # right-handed people, # left-handed people
X,   100, 90
Y,   80,  75
Z,   90,  120
...

How can we estimate the true distribution of right-handed people vs left-handed people in each city from this data knowing that there is a bias in the samplng method? I am looking for both Frequentist and Bayesian solutions.

Answer 1

First, notice that your estimate depends on how much your sample is biased. For example, say you are interested in average height in the population and you samples only males and no females. This sample would certainly lead you to biased results since there is a difference in height of both genders. So in this kind of cases nothing could be done, since you know nothing about the cases that were not sampled.

If the bias are known you can apply survey weights to correct for those biases. The general idea is that you assign some weights to your observations such that the underrepresented cases are given greater importance, while overrepresented cases lower importance. Weighted mean is a simple example of this approach.

If you have no idea about sample bias, one approach would be to use bootstrap method that was designed for estimating the unknown distributions. In bootstrap you create multiple bootstrap samples by sampling with replacement $N$ cases out of $N$ observations in your dataset in the same manner as you sampled your data from the population. This lets you approximate the (unknown) distribution underlying your data by re-creating it from the sample. See this thread to get better understanding on how bootstrap works, check the books by Efron and Tibshirani (1993), and Davison and Hinkley (2009) and here you will find an example. With bootstrap, if you knew more about your sample, you could also use weights to correct for bias. The method however is not perfect and cannot be used blindly.

Bayesian approach for this case would be also pretty simple: if $x_i$ are counts or right-handed people in $i = 1,...,N$ cities, where $n_i$ are total populations of the cities, then you could assume Binomial distribution for the counts

$$x_i \sim Binomial(n_i, p)$$

where for parameter $p$ you could assume a "flat", $Uniform(0, 1)$ prior (uninformative), or informative Beta prior if you have some prior assumptions about the proportion. Using informative prior enables you to include out-of-sample information in your model (e.g. about general number of right-handers in the population), and so possibly to correct for the bias, what would be not possible in non-Bayesian approach. The easiest way to estimate this kind of model would be with JAGS or Stan.