Is non-sampling error included in hypothesis testing assumptions?

Question

I understand that, in hypothesis testing, we are dealing with sampling error. We calculate the statistic from the sample data and test what the probability of getting that result is when the null hypothesis is true. But is the hypothesis testing taking also into account non-sampling errors?

I'm asking this because I had a discussion with a co-worker about if I have to take into account the p-value of a correlation test if I'm taking the whole population as "sample" (I have even made a question about that here, Rho Spearman correlation test interpretation for sample equal to population ). First I thought that since we are calculating the correlation with the larger sample possible, obviously there is no sampling error, so there is no necessity for a significance test, but my co worker said that we should take the p-value always into account because it is testing the hypothesis against non sampling error.

Is that true?

Answer 1

Your colleague is wrong and you are right. There is no sense in carrying out statistical tests when you are working with population statistics or a census. If your analysis includes every observation, there is no sampling error -- the summary statistics you calculate do not have any uncertainty. For example, if you were interested in the average age of high school students in a specific school, and you took a sample of 10 students, then it would make sense to compute a confidence interval and compute $p$-values to indicate the uncertainty of your sample. If you instead conducted a census of the school (and did not have missing values), and the average age was, say, 16, then this is the actual population value and there would be no uncertainty behind this summary number. There would be no need to compute confidence intervals because you are 100% certain that in repeated sampling of all students, you'd get the same value.

The same goes for correlation coefficient or any other statistics you could compute. If you have calculated a population statistic like $\rho$, then if it is any value other than zero, then you'd reject your null hypothesis that the correlation is equal to zero. The next question you'd have to ask yourself, is: is the value of $\rho$ of practical difference to you? For example if your value of $\rho$ is $0.000001$, is that practically the same as 0 in the context of your problem?

$p$-values and confidence intervals only take into account sampling error.

It's quite common (and somewhat frustrating when I see it) to see students of statistics so trained at the rote computations learned in their statistical classes that they often forget that there's no need to carry out the complex sample statistical analyses for population statistics. Inferential statistics only apply when you need to make, well, inferences, about something unknown. So, it's always a wise step to think about the population and sample before beginning any analysis problem. If you have a dataset of your population, there is no need to compute any $p$-values, confidence intervals, or any other inferential statistics.