# What is the right way to decide the sample size for this survey?

My country Costa Rica is having elections and I am intrigued about one of the polls I've seen. The voting population is 3322329, so they took a sample of 3300 people for an error margin of 1.7% and a confidence level of 95%.

Everything good so far, where I am having doubts is how then chose their sample. The country is divided in 7 provinces, so they selected the percentage of voters from each province and applied the survey on each province to the corresponding number of people.

To me this is effectively running 7 different surveys. In San José for example, that has 1114779 voters or 33.9% of the total, they performed 1119 surveys.

Of a total of 1114779 people, a sample of 1119 at a 95% confidence level, it gives me an error rate of 2.93%, not the 1.7% of the total sample. Am I thinking of this right?

Say you take a sample of size $$n_j$$ in area $$j$$, and register the number of votes $$x_{ij}$$ for party $$i$$ in area $$j$$ then this will be binomial distributed

(or multinomial if you consider multiple groups at the same time, but for the expression of the variance it does not matter)

$$x_{ij} \sim B(n_j,p_{ij})$$

where $$p_{ij}$$ is the fraction of votes for party $$i$$ in region $$j$$

(here I will assume that these $$p_{ij}$$ is known to make calculations easier, and just demonstrate the distribution of the outcome, but in practice you will express the error based on obtained estimates $$\hat{p}_{ij}$$ which requires a correction for the calculation of the error margins, although with large numbers this correction does not matter much).

The variance of the observations will be:

$$\text{Var}(\hat{x}_{ij}) = n_jp_{ij}(1-{p_{ij}})$$

and

$$\text{Var}(\hat{p}_{ij}) = \text{Var}(\frac{\hat{x}_{ij}}{n_j}) = \frac{p_{ij}(1-{p_{ij}})}{n_j}$$

Now you will use a weighted combination of multiple estimates, $$\hat{p}_{ij} = \frac{\hat{x}_{ij}}{n_j}$$ , from different regions to estimate a total $$\hat{p}_{i} = \sum f_j \hat{p}_{ij}$$ for the entire population (where $$f_j$$ is the fraction of the entire population that lives in the region $$j$$).

which has variance:

$$\text{Var}(\hat{p}_{i}) = \sum_{\forall j} f_j^2\text{Var}(\hat{p}_{ij}) = \sum_{\forall j} \frac{f_j^2}{n_j} p_{ij}(1-{p_{ij}})$$

# To the point 1

So note that the variance in the weighted mean

$$\text{Var}(\hat{p}_{i}) = \sum_{\forall j} f_j^2\text{Var}(\hat{p}_{ij})$$

is smaller than the variance of a standard arithmetic mean

$$\overline{\text{Var}(\hat{p}_{ij})} = \sum_{\forall j} f_j\text{Var}(\hat{p}_{ij})$$

So you may get such values as error rates of 2.93%, but that is for the regions, not for the total. The error rate for the total a weighted a sum of the values in the regions, will be smaller than the average of the individual error rates. See also https://en.wikipedia.org/wiki/Variance#Basic_properties

Intuitively you can imagine that if you combine the 7 separate test that the errors (positive and negative) will even out a little bit such that the total error will be relatively smaller.

# To the point 2

The reason to distribute the sample over the population is because, for a given total sample size $$n = \sum n_j$$ one can make a most optimal distribution of the $$n_j$$ that results in a lower variance.

So we wish to minimize

$$\text{Var}(\hat{p}_{i}) = \sum_{\forall j} f_j^2\text{Var}(\hat{p}_{ij}) = \sum_{\forall j} \frac{f_j^2}{n_j} p_{ij}(1-{p_{ij}}) \qquad \text{with \sum n_j = n}$$

which is not straight forward since the term $$p_{ij}(1-{p_{ij}})$$ is unknown. But if we imagine it is known then we get (using a Lagrangian method, sorry I could not find a reference for this particular example)

$$n_j \propto f_j \sqrt{p_{ij}(1-{p_{ij}})}$$

where $$\propto$$ stands for proportional to. You'd have to find a normalization constant such that you end up with $$\sum n_j = n$$.

When we ignore the $$\sqrt{p_{ij}(1-{p_{ij}})}$$, which we do not know (and when prior estimates of variances in $$f_j$$ are larger than variances in $$p_j$$ then this is not such bad plan), then you might imagine it could still be useful to ensure (fix) $$n_j = f_j n$$ rather than using some random process let determine the $$n_j$$ (which would result in a likely less optimal distribution of the $$n_j$$, because it is likely not a homogeneous distribution like exactly $$n_j = f_j n$$, this is especially important when you factor in even more groups than just the seven regions/areas, like age, gender, education, etc. )

While it would be appropriate to examine each subset for each province, it would only provide an estimate for that province.  In such a scenario, you would be correct in calculating the margin of error specific to the sample obtained from that locale.  However, the survey researchers (if they followed a conventional protocol) would have stratified their sample to obtain a random sample that had comparable proportions to the entire nation.  While this would not be a true simple random sample, it would be a representative random sample, and the more conventional margin of error calculations would generally apply.

You say "To me this is effectively running 7 different surveys" - this is good intuition. When we stratify in surveys we independently select within each stratum so we are in effect running separate surveys. Also within each of these stratum level surveys we'll tend to have wider margins of error than across the whole survey.

To roughly see why this is, firstly I'll say that even though I don't know much about politics in Costa Rica, it's probably fair to assume that people within each province are more similar to each other than people between provinces (even if just a little). This means that if our random sample over-represented one particular province then the survey statistics would tend to differ from the population statistics. If we instead control the amount of sample from each province then we don't have to worry about accidentally picking a sample that over-represented a particular stratum, and hence would likely vary from the population statistics.

Because stratification makes it impossible to select some of the samples that may give the most extreme statistics, the margin of error that comes from stratified sampling tends to be less than a simple random sample.