Why is the design effect in most sample studies taken as 1.25?

Question

Why is the design effect in most sample studies taken as 1.25? Who and how was it calculated first of all?

Answer 1

We often don't have the luxury of taking a simple random sample from a population, or sometimes, collecting a simple random sample isn't the best approach to collecting the data we need to answer our research question. Frequently, when we conduct a survey (or study): we stratify our sample, apply sampling weights, sample from within clusters, or apply some other design element. Most (but not all) departures from a simple random sample introduce additional variance, and we have to account for this additional variance when we calculate our standard errors.

The classic design effect proposed by Kish (sometimes written 'Deft') compares the design we actually implement to a hypothetical simple random sample design. A design effect greater than one indicates that the design is less efficient than a hypothetical simple random sample from the same population would have been. If a simple random sample would have given us an estimate with a $\pm5\%$ margin of error, but our study has a design effect of 2.0, we would expect a margin of error for this estimate to be closer to $\pm10\%$.

So$\dots$ "Why is the design effect in most sample studies taken as 1.25?" I don't know that I would agree that this is the case; I often see studies with greater or lesser design effects. But, there are a few possibilities to consider:

• Strictly speaking, the survey itself or the study itself doesn't have a design effect, unless only one question was asked or only one measure was collected. Rather each estimate has its own design effect. But oftentimes, a design effect is reported for a whole study or whole survey. Unless the author specifies, it can be unclear what this number means; this is also often true when a 'margin of error' is reported for a survey. It could be: an average design effect across multiple items, the maximum design effect for any item, based on what the authors expect given previous similar studies, be based on assumptions in the design, or worst case scenario, it could just "sound about right."

• We can't alway estimate the design effect before we conduct the study. For example, for a survey with a cluster design, we cannot estimate the design effect until after we have collected the data and obtained a measure of intra-class correlation for clusters. So in a discussion of sampling, you might often see language like, "assuming a design effect of 1.25," which means that the design effect could not be estimated beforehand but based on references to other similar studies, the authors expect the design to be slightly less efficient than a simple random sample would be. Usually when we assume a design effect greater than one, this means that we will seek to obtain a sample larger than we would have under the assumption of simple random sampling.

• Finally, some researchers assume a design effect to be "on the safe side" or to provide more "conservative estimates." This is usually done by some rule of thumb, which probably varies by field (maybe in your field, 1.25 is a rule of thumb?), but in short, when a researcher is doing this, he or she is acknowledging that there may be additional sampling error from unknown or undetected sources.

As for how a design effect is calculated, the classic design effect given by Kish in 1965 is: $$ Deft = 1 + (m-1)\rho $$ Where $\rho$ is a measure of intraclass correlation, and $m$ is the size of clusters.

However, the way this is used today most often incorporates a number of extensions for design elements in addition to clustering. This is nicely discussed in this paper (no paywall). For a rather conventional stratified cluster design, a standard way to calculate a total design effect for an estimate would be to calculate the product of (a) a clustering design effect, (b) a stratification design effect, and (c) a weighting design effect. (Equations given in linked paper).