# Are significance tests only about sampling error or other forms of variation also?

When we study differences between a treatment and control group in an experimental setting, we want to test whether those differences are big enough that we can conclude they are not simply due to chance.

(Lacking official terminology for the some concepts below [especially #2], please either grant me a little creative license or suggest the appropriate technical terms)

Differences due simply to chance, i.e., stochastic variation, could come from:

1. Sampling error, if the sample is smaller than the population.
2. Inherent error (or parallel universe error) -- There is some inherent unmeasurable randomness in the manifestation of an outcome in any given individual. There is a broader universe of possible outcomes than what we actually observe. Even if we measure the whole population, there is some fundamental level of variation where these observations could have been different, even if nothing meaningful or causal changed.
3. Measurement error -- Given imperfect questions, questioning, or responses, we are not measuring exactly the concept we are aiming for.
4. Temporal variation -- the moment at which we interview someone may lead to variation in answers even if they answer with perfect accuracy. e.g., "how much money did you spend on good X in the past 7 days?" This would vary depending on which day we interview the person.

For each individual, we observe an outcome with some noise.

$y_i = X_i \beta + \epsilon_i$

But the noise is due to a mix of the factors above.

In my specific case, my team has surveyed 76% of our entire population of interest (due to 24% non-response rate). If we limit our scope conditions to the people who responded to the survey, then we have surveyed 100% and we have no sampling error. My colleague argues that we do not need significance tests. For example, if we observe that group A has $\bar{X_1} = 40$ and group B has $\bar{X_1} = 41$, we can infer that this is a meaningful difference.

My perspective is that we should incorporate the variance of the measures because the magnitude of difference and variance of the data can still tell us about whether we are observing systematic variation between groups as opposed to stochastic variation. Even though if we had measured our entire "real" population of interest, Due to stochastic variation from $\epsilon_i$ (or if $\epsilon_i$ is correlated with group assignment) we might observe differences between the two groups that we cannot interpret as "substantive," or "an effect of treatment" in an experimental setting.

Is my colleague correct that significance tests are exclusively about sampling error?

How should a researcher test for stochastic variation other than sampling error?

• You seem to conflate "meaningful difference" with "statistically significant difference" but they are not the same. A small difference can be statistically significant but not meaningful. In your example, is a score of 40 meaningfully different than a score of 41? Also, you seem to focus only on means. If might be that you have differences in variances that are important to know about. As to your error types numbered 3 and 4, they should be constant across groups and therefore contribute only error to the data. I will let others answer your other questions. – Joel W. Jul 16 '14 at 16:05
• +1 @JoelW. Good observation that (3) and (4) would be constant across groups in expectation. In addition to your valid distinction between "meaningful" and "statistically significant difference," there is a third concept: "causal effect." If we saw the scores 40 and 41, can we say that this difference is due to treatment? In the experimental setting with an entire population surveyed and well-balanced pre-treatment covariates, if my colleague is correct that we have no need to test for stochastic variation (e.g., through significance tests), then we can say that effect of treatment is 1. – Dr. Beeblebrox Jul 16 '14 at 16:49
• You say you have surveyed your entire population of interest, but also refer to possible temporal variation. This suggests that it might be helpful to define the population of interest not only in terms of which individuals belong but also in terms of temporal limits, implying that only a sample will have been surveyed, and re-interpreting temporal variation as a form of stochastic variation. If you were really interested only in a set of individuals, each at a defined point in time, then temporal variation would not be relevant. – Adam Bailey Jul 16 '14 at 17:53