When should you choose Stratified sampling over random sampling?

Question

Is there a general rule to follow when deciding when it's best to use one over the other?

An example I was looking at was the following:

An example might be to determine the proportions of defective products being assembled in a factory. In this case sampling may be stratified by production lines, factory, etc.

Answer 1

The other answer is wrong, unfortunately. Actually, that should be obvious, otherwise no statistician would ever bother to do SRS and they would happily stratify on utterly useless variables- why not?

It is true that the sum of squared error inside each stratum will drop (i.e. $\sum (x_i - {\overline{x}})^2$ will drop virtually always). This is similar to the fact that adding new variables to a regression will just about always lower the R-squared even if the variable is complete noise. However, the value of $s^2$ also involves dividing by $n_1 - 1$ and not by $n_i$. In other words, if you have $k$ strata, you lose $k-1$ degrees of freedom. So your estimated variances might rise if the stratification variable is sufficiently useless. (The effect is most serious if sample sizes are small in some strata. In the extreme, if they drop to $n_i = 1$, you can't even estimated the standard error.)

Crudely, a fast way to tell if stratification would help is to run an ANOVA on the stratification variable. If it's significant (or, at least, the ADJUSTED R-squared rises), the stratification might help.

In practice, I tell researchers that as long as they have a reasonable sample size available from each strata, and the stratification variable makes sense (they are sure that means of one strata are 'significantly different' from those of other strata), then stratify.

Side note: While lowering variances is the usual reason to stratify, there are others. First, if you want to guarantee sufficient sample size in each stratum so that you can make separate inferences on each one, you should stratify. Second, if costs vary greatly from one stratum to another, you can stratify to optimize costs. Finally, if variability is known to be much higher in some strata than others, you can use stratification (by increasing sample size in the most variable groups) to lower your se. However, if costs and variables don't distinguish your strata, you can definitely get a wider confidence interval if you stratify on an unhelpful variable.

I'll illustrate with an exact computation: I'll start with the population $\lbrace 100, 150, 50, 101, 151, 51\rbrace$. First I'll enumerate the EXACT sampling distribution of the means of all possible Simple Random Samples (SRS) of size n = 4 from the population. Then I'll break this into two strata, each of size three. I'll enumerate all possible means of samples based on an SRS of size 2 from each stratum.

Finally, I'll compute the exact 'population' variance (i.e. sigma squared) of each statistic.

> #The usual unbiased estimator of the mean, 
> #for a SRS of size n = 4 is the mean.  I’ll find 
> #its exact sample distribution.
> 
> pop = c(100, 150, 50, 101, 151, 51)
> require(gtools)
> subsets = combinations(n=6, r=4)
> subsets[] = sapply(subsets, FUN=function(x){pop[x]})
> samp_dist = rowMeans(subsets)  # exact sampling distribution of sample means
> samp_dist
 [1] 100.25 112.75  87.75 125.50 100.50 113.00 100.50  75.50  88.00 100.75
[11] 113.00  88.00 100.50 113.25  88.25
> mu = mean(samp_dist)
> sigma2_sampling_dist = sum((samp_dist - mu)^2)/length(samp_dist)
>    # Note:  divided by n because this is a true variance (on a census), not an estimator
> sigma2_sampling_dist
[1] 166.6917
> 
> 
> 
> #Now consider stratification into two strata:
> 
> st1 = c(100, 150, 50)
> st2 = c(101, 151, 51)
>   # Take a SRS of size two from each stratum.  I won’t bother with
>   # combinations, as there aren’t many possible samples.  Then
>   # take the mean of each, followed by the average of these two means.
> sampling_dist1 = c(mean(c(100,150)),mean(c(100,50)),mean(c(150,50)))
> sampling_dist2 = c(mean(c(101, 151)), mean(c(151, 51)), mean(c(101,51)))
> samp_dist2 = rowMeans(cbind(rep(sampling_dist1, each=3), 
+                                       rep(sampling_dist2,times=3)))
> samp_dist2
[1] 125.5 113.0 100.5 100.5  88.0  75.5 113.0 100.5  88.0
> mu2 = mean(samp_dist2)
> sigma2_sampling_dist2 = sum((samp_dist2 - mu2)^2)/length(samp_dist2)
> sigma2_sampling_dist2
[1] 208.3333

Note that the true variance of the stratification estimator is much larger than the variance of the simple random sample estimator. By the way, if I repeat this for the population $\lbrace 100, 150, 50, 170, 220, 120\rbrace$, where the strata are considerably different, I get the stratification estimator working better:

exact variance of SRS estimator: 289.1667

exact variance of stratification estimator: 208.3333

Actually, it probably would have been easier to just prove this than give an example. But this shows that stratification can fail to give a lower variance estimator. Note that this example is extreme in that the sample sizes are small.

Answer 2

I'll make several statements and then prove them mathematically, in case you're interested. If you want a quick summary, I'll provide one at the end.

First of all, both simple random sampling (SRS) and stratified sampling will provide you with an unbiased estimator of population mean $\mu$.

Proof 1:

Denote by $\bar{x}_{SRS}$ sample mean for SRS and $\bar{x}_{St}$ sample mean for stratified sampling.

$\bar{x}_{SRS}$ is an unbiased estimator for $\mu$

$$ \begin{aligned} E[\bar{x}_{SRS}] = \frac{1}{N} X_1 + ... + \frac{1}{N} X_N = \bar{X}_{SRS} = \mu \end{aligned} $$

Taking the previous and applying it, given $L$ strata, $\bar{x}_{St}$ is an unbiased estimator for $\mu$

$$ \begin{aligned} E[\bar{x}_{St}] &= E[\sum^L_{i=1} W_i \bar{x}_i] \\ &= \sum^L_{i=1} W_i E(\bar{x}_i) \\ &= \sum^L_{i=1} W_i \bar{X}_i \\ &= \frac{N_1 \bar{X}_1 + ... + N_L \bar{X}_L}{N} \\ &= \frac{\tau_1 + ... \tau_L}{N} \\ &= \bar{X} \\ &= \mu \end{aligned} $$

End proof

Since both sampling schemes give you an unbiased estimation, either is fine to use. However, the variances are not equal, and thus we can define conditions under which it is optimal to perform stratified sampling.

Recall that $W$ is the weight per group ie. $\frac{n_h}{N}$. $$ \begin{aligned} V_{prop} &= \sum^L_{h=1} \frac{w^2_h s^2_h}{n W_h} (\frac{N w_h - n W_h}{N W_h }) \\ &= ( \frac{1}{n} \sum^L_{h = 1} w_h s^2_h) \frac{N-n}{N} \\ &= \frac{N-n}{Nn} \sum^L_{h=1} w_h s^2_h \end{aligned} $$

Recall that

$$ \begin{aligned} V_{ran} &= \frac{S^2}{n} (\frac{N-n}{N}) \\ V_{prop} &= \frac{N-n}{Nn} \sum^L_{h=1} W_h S^2_h \\ V_{opt} &= \frac{1}{n} (\sum^L_{h=1} W_h S_h)^2 - \frac{1}{N} \sum^L_{h=1} W_h S^2_h \end{aligned} $$

Recall that $W$ is the weight per group ie. $\frac{n_h}{N}$

$$ \begin{aligned} S^2 &= \frac{1}{N-1} \sum^N_{i=1} (Y_i - \bar{Y})^2 \\ (N-1) S^2 &= \sum^N_{i=1} (Y_i - \bar{Y})^2 \\ &= \sum^L_{h=1} \sum^{N_h}_{i=1} (Y_{hi} - \bar{Y})^2 \\ &= (Y_{hi} - \bar{Y_h} + \bar{Y_h} - \bar{Y})^2 \\ &= \sum^L_{h=1} \sum^{N_h}{i=1} (Y_{hi} - \bar{Y}_h)^2 + \sum^L_{h=1} \sum^{N_h}_{i=1} (\bar{Y}_h - \bar{Y})^2 + 2 \sum^L_{h=1} \sum^{N_h}_{i=1} (Y_{hi} - \bar{Y}_h)(\bar{Y}_h - \bar{Y} \end{aligned} $$

Recall that subtracting the mean from a series of data is always 0. Since $\sum^{N_h}_{i=1} (Y_{hi} - \bar{Y}_h) = 0$, the third term disappears.

$$ \begin{aligned} S^2_h &= \frac{1}{N_h -1} \sum^{N_h}_{i=1} (Y_{hi} - \bar{y}_h)^2 \\ (N-1) S^2 &= \sum^L_{h=1} (N_h -1) S^2_h + \sum^L_{h=1} N_h (\bar{Y}_h - \bar{Y})^2 \end{aligned}$$

Note that $f = \frac{n}{N}$ aka finite population correction.**

$$ \begin{aligned} V_{ran} ( \bar{y}) &= \frac{1 - f}{n} S^2 \\ &\approx \frac{1-f}{n} \sum^L_{h=1} W_h S^2_h + \frac{1-f}{n} \sum^L_{h=1} W_h (\bar{Y}_h \bar{Y})^2 \\ V_{SRS} - V_{St} &= \frac{1-f}{n} \sum^L_{h=1} W_h S^2_h + \frac{1-f}{n} \sum W_h (\bar{Y}_h - \bar{y})^2 - \frac{1}{n} (\sum^L_{h=1} W-h_h S_h)^2 + \frac{1}{N} \sum^L_{h=1} W_h S^2_h \\ &= \frac{1}{n} \sum^L_{h=1} W_h S^2_h - \frac{1}{N} \sum^L_{h=1} W_h S^2_h + \frac{1-f}{n} \sum W_h (\bar{Y}_h - \bar{Y})^2 + \frac{1}{N} \sum^L_{h=1} W_h S^2_h - \frac{1}{n} (\sum^L_{h=1} W_h S_h)^2 \\ &= \frac{1}{n} \sum^L_{h=1} W_h S^2_h - (\sum^L_{h=1} W_h S_h)^2) + \sum^L_{h=1} W_h (\bar{Y}_h - \bar{Y})^2 \\ &= \frac{1}{n} \sum^L_{h=1} W_h (S_h \bar{S})^2 + \sum^L_{h=1} W_h (\bar{Y}_h - \bar{Y})^2 \\ V_{ran} - V_{prop} &= \frac{1-f}{n} \sum^L_{h=1} W_h S^2_h + \frac{1-f}{n} \sum^L_{h=1} W_h (\bar{Y}_h - \bar{Y})^2 - \frac{1}{n} W_h S^2_h + \frac{1}{N} W_h S^2_h \\ &= \frac{1-f}{n} \sum^L_{h=1} W_h (\bar{Y}_h - \bar{Y})^2 \end{aligned} $$

Interpretation:

We look at two kinds of stratified sampling schemes, proportion and optimum (Neymar Allocation) and show that both are better than simple random sampling. The proportional allocation method performs better than SRS when the following is maximized:

$$ \frac{1-f}{n} \sum^L_{h=1} W_h (\bar{Y}_h - \bar{Y})^2 $$

The only control we have over this expression is the difference between $\bar{Y}_h$ and $\bar{Y}$. This means that if you have strata that have means far from the grand mean, then proportional allocation will give you a smaller variance, and thus an optimal, better, sample.

The second kind, Neymar or optimal allocation, wants us to maximize the following in order to have the biggest difference, and thus the smallest variance:

$$ \frac{1}{n} \sum^L_{h=1} W_h (S_h - \bar{S})^2 + \sum^L_{h=1} W_h (\bar{Y}_h - \bar{Y})^2 $$

This gives us an additional term to the proportional allocation above. Thus, optimal allocation is better than proportional allocation because if the standard deviations of the groups are different than the grand standard deviation, then this term is bigger than the one above. There is no way that it is smaller. Thus, as a summary:

$$ V_{opt} (\bar{y}_{st}) \leq V_{prop} (\bar{y}_{st}) \leq V_{SRS} (\bar{y}_{SRS}) $$

Note that the above formulations hold when we assume $\frac{1}{N} \approx \frac{1}{N_i} \overset{.}{=} 0$ and assume that $\frac{N_h - 1}{N-1} \approx \frac{N_h}{N}$. When this assumption is not made, the above is slightly more complex, but still follows.

I've probably made some mistakes and some typos; I'll fix them when I have a little more time, but hopefully the general idea comes across.

TL;DR

Stratification is always better, assuming equal costs of sampling each strata. It's best when the mean and standard deviation of your strata are really different than your grand mean and standard deviation.

---

References:

Elementary Survey Sampling 7th Edition, Richard L. Scheaffer (Author), III William Mendenhall (Author), R. Lyman Ott (Author), Kenneth G. Gerow (Author), ISBN-13: 978-0840053619

Answer 3

I've taken many samples, large and small, simple and complex, over the years. My conclusion: Simple random sampling (SRS) alone is almost never the choice for a real-world problem.

On the other hand the theory of SRS is important, because it underlies the theory of other techniques.

The alternatives to SRS: stratified sampling, systematic sampling, in some instances, unequal probability sampling, or a combination of these. It is okay to take an SRS within strata.

In my comments, I quoted Cochran as saying stratified sampling isn't always more precise than SRS. However increased precision is not the only, or even the main, reason for choosing a stratifed design.

Reasons to stratify

Look for stratifying factors for at five reasons (Lohr (2009) p. 74; Valliant, Dever, & Kreuter, 2013, p. 44):

1. To avoid selecting a sample that badly misrepresents the population. I've seen many instances of such SRSs. In some, reweighting was a partial fix. In others, no recovery was possible. One such was the object of a question to Statalist. A senior public health official wanted to estimate characteristics of an epidemic by studying tf patients who attended medical clinics during that time. There were 40 clinics in the city, and 10 were drawn by SRS. Unfortunately, the 10 did not include the two very large hospital clinics in the city, which between them saw over 30% of all outpatients, usually the sickest. This bias made the sample useless for the satisfying its original purpose. At a minimum, the two large hospital clinics should have been selected with certainty before taking the simple random sample.

2. Closely related: stratify to "cover" the entire population. (This is also a reason to do systematic sampling.)

3. To guarantee a minimum sample size for group that are going receive separate analyses. For a study of occupational health and safety in California farms, for example, farms were stratified by size and major crop.

4. To control costs. Example: charts were to be abstracted in a sample of California hospitals. Rural hospitals were placed in a separate stratum and sampled at a lower rate than urban hospitals. Why? Experienced abstractors lived in urban areas and could study 1-2 hospitals per day, then go home at night. To study a rural hospital took one abstractor two days, including travel, and incurred the cost of an overnight stay.

5. To improve sample efficiency (i.e. get smaller standard errors) by grouping together observations with similar means and variances. Some national surveys stratify as finely as possible and draw a SRS with $n= 1$ unit from each stratum. Because a minimum of $n = 2$ observations per stratum is needed to compute standard errors, such designs are analyzed by combining neighboring strata. The "true" standard errors for the design are then likely to be smaller than the estimated standard errors, clearly a good thing.

6. To sample with probability approximately proportional to size.

Systematic sampling

Many frames have a natural ordering, for example date of event. Systematic samples capture the natural stratification contained in this ordering.

References

Lohr, Sharon L. 2009. Sampling: Design and Analysis. Boston, MA: Cengage Brooks/Cole.

Valliant, Richard, Jill A. Dever, and Frauke Kreuter. 2013. Practical Tools for Designing and Weighting Survey Samples. Statistics for Social and Behavioral Sciences. Springer.