When we study differences between a treatment and control group in an experimental setting, we want to test whether those differences are big enough that we can conclude they are not simply due to chance.
(Lacking official terminology for the some concepts below [especially #2], please either grant me a little creative license or suggest the appropriate technical terms)
Differences due simply to chance, i.e., stochastic variation, could come from:
- Sampling error, if the sample is smaller than the population.
- Inherent error (or parallel universe error) -- There is some inherent unmeasurable randomness in the manifestation of an outcome in any given individual. There is a broader universe of possible outcomes than what we actually observe. Even if we measure the whole population, there is some fundamental level of variation where these observations could have been different, even if nothing meaningful or causal changed.
- Measurement error -- Given imperfect questions, questioning, or responses, we are not measuring exactly the concept we are aiming for.
- Temporal variation -- the moment at which we interview someone may lead to variation in answers even if they answer with perfect accuracy. e.g., "how much money did you spend on good X in the past 7 days?" This would vary depending on which day we interview the person.
For each individual, we observe an outcome with some noise.
$y_i = X_i \beta + \epsilon_i$
But the noise is due to a mix of the factors above.
In my specific case, my team has surveyed 76% of our entire population of interest (due to 24% non-response rate). If we limit our scope conditions to the people who responded to the survey, then we have surveyed 100% and we have no sampling error. My colleague argues that we do not need significance tests. For example, if we observe that group A has $\bar{X_1} = 40$ and group B has $\bar{X_1} = 41$, we can infer that this is a meaningful difference.
My perspective is that we should incorporate the variance of the measures because the magnitude of difference and variance of the data can still tell us about whether we are observing systematic variation between groups as opposed to stochastic variation. Even though if we had measured our entire "real" population of interest, Due to stochastic variation from $\epsilon_i$ (or if $\epsilon_i$ is correlated with group assignment) we might observe differences between the two groups that we cannot interpret as "substantive," or "an effect of treatment" in an experimental setting.
Is my colleague correct that significance tests are exclusively about sampling error?
How should a researcher test for stochastic variation other than sampling error?