Are significance tests only about sampling error or other forms of variation also?

Question

When we study differences between a treatment and control group in an experimental setting, we want to test whether those differences are big enough that we can conclude they are not simply due to chance.

(Lacking official terminology for the some concepts below [especially #2], please either grant me a little creative license or suggest the appropriate technical terms)

Differences due simply to chance, i.e., stochastic variation, could come from:

1. Sampling error, if the sample is smaller than the population.
2. Inherent error (or parallel universe error) -- There is some inherent unmeasurable randomness in the manifestation of an outcome in any given individual. There is a broader universe of possible outcomes than what we actually observe. Even if we measure the whole population, there is some fundamental level of variation where these observations could have been different, even if nothing meaningful or causal changed.
3. Measurement error -- Given imperfect questions, questioning, or responses, we are not measuring exactly the concept we are aiming for.
4. Temporal variation -- the moment at which we interview someone may lead to variation in answers even if they answer with perfect accuracy. e.g., "how much money did you spend on good X in the past 7 days?" This would vary depending on which day we interview the person.

For each individual, we observe an outcome with some noise.

$y_i = X_i \beta + \epsilon_i$

But the noise is due to a mix of the factors above.

In my specific case, my team has surveyed 76% of our entire population of interest (due to 24% non-response rate). If we limit our scope conditions to the people who responded to the survey, then we have surveyed 100% and we have no sampling error. My colleague argues that we do not need significance tests. For example, if we observe that group A has $\bar{X_1} = 40$ and group B has $\bar{X_1} = 41$, we can infer that this is a meaningful difference.

My perspective is that we should incorporate the variance of the measures because the magnitude of difference and variance of the data can still tell us about whether we are observing systematic variation between groups as opposed to stochastic variation. Even though if we had measured our entire "real" population of interest, Due to stochastic variation from $\epsilon_i$ (or if $\epsilon_i$ is correlated with group assignment) we might observe differences between the two groups that we cannot interpret as "substantive," or "an effect of treatment" in an experimental setting.

Is my colleague correct that significance tests are exclusively about sampling error?

How should a researcher test for stochastic variation other than sampling error?

Answer 1

Based on conversations I've had with colleagues, it seems the consensus is to include conventional confidence intervals. Although the reason depends on the context, it seems to be considered best practice to be conservative by reporting confidence intervals. If anything the "correct" standard errors will be smaller than the ones reported using conventional SEs.

Here's a superstar econometrics paper by Abadie, Athey, Imbens, Woolridge that argues that one should report standard errors even when one has full data (aka state-level data or census-level data): http://www.tinbergen.nl/wp-content/uploads/2014/01/220114_Imbens.pdf

The abstract from the Abadie et al. paper:

When a researcher estimates a regression function with state level data, why are there standard errors that differ from zero? Clearly the researcher has information on the entire population of states. Nevertheless researchers typically report conventional robust stan- dard errors, formally justified by viewing the sample as a random sample from a large population. In this paper we investigate the justification for positive standard errors in cases where the researcher estimates regression functions with data from the entire popu- lation. We take the perspective that the regression function is intended to capture causal effects, and that standard errors can be justified using a generalization of randomization inference. We show that these randomization-based standard errors in some cases agree with the conventional robust standard errors, and in other cases are smaller than the conventional ones.