# Statistical inference when the sample “is” the population

Imagine you have to do reporting on the numbers of candidates who yearly take a given test. It seems rather difficult to infer the observed % of success, for instance, on a wider population due to the specifity of the target population. So you may consider that these data represent the whole population.

Are results of tests indicating that the proportions of males and females are different really correct? Does a test comparing observed and theoretical proportions appear to be a correct one, since you consider a whole population (and not a sample)?

-
thanks for change the title Jeromy. better this way. –  pbneau Sep 14 '10 at 19:29

There may be varying opinions on this, but I would treat the population data as a sample and assume a hypothetical population, then make inferences in the usual way. One way to think about this is that there is an underlying data generating process responsible for the collected data, the "population" distribution.

In your particular case, this might make even more sense since you will have cohorts in the future. Then your population is really cohorts who take the test even in the future. In this way, you could account for time based variations if you have data for more than a year, or try to account for latent factors through your error model. In short, you can develop richer models with greater explanatory power.

-
Just came across this post from A Gelman, How does statistical analysis differ when analyzing the entire population rather than a sample?, j.mp/cZ1WSI. A good starting point about diverging opinions on the concept of a "super-population". –  chl Oct 31 '10 at 17:20
@chl: interesting -- reminds me that Gelman had a discussion of finite/super population inference being comparable to fixed-/random-effects in his paper on ANOVA [ stat.columbia.edu/~gelman/research/published/econanova3.pdf ]. –  ars Oct 31 '10 at 17:53
+1 I just came back to this again (through google). I think that your answer is spot on. –  Shane Oct 31 '10 at 23:37

Actually, if you're really positive you have the whole population, there's even no need to go into statistics. Then you know exactly how big the difference is, and there is no reason whatsoever to test it any more. A classical mistake is using statistical significance as "relevant" significance. If you sampled the population, the difference is what it is.

On the other hand, if you reformulate your hypothesis, then the candidates can be seen as a sample of possible candidates, which would allow for statistical testing. In this case, you'd test in general whether male and female differ on the test at hand.

As ars said, you can use tests of multiple years and add time as a random factor. But if your interest really is in the differences between these candidates on this particular test, you cannot use the generalization and testing is senseless.

-

Traditionally, statistical inference is taught in the context of probability samples and the nature of sampling error. This model is the basis for the test of significance. However, there are other ways to model systematic departures from chance and it turns out that our parametric (sampling based) tests tend to be good approximations of these alternatives.

Parametric tests of hypotheses rely on sampling theory to produce estimates of likely error. If a sample of a given size is taken from a population, knowledge of the systematic nature of sampling makes testing and confidence intervals meaningful. With a population, sampling theory is simply not relevant and tests are not meaningful in the traditional sense. Inference is useless, there is nothing to infer to, there is just the thing...the parameter itself.

Some get around this by appealing to super-populations that the current census represents. I find these appeals unconvincing--parametric tests are premised on probability sampling and its characteristics. A population at a given time may be a sample of a larger population over time and place. However, I don't see any way that one could legitimately argue that this is a random (or more generally any form form of a probability) sample. Without a probability sample, sampling theory and the traditional logic of testing simply do not apply. You may just as well test on the basis of a convenience sample.

Clearly, to accept testing when using a population, we need to dispense with the basis of those tests in sampling procedures. One way to do this is to recognize the close connection between our sample-theoretic tests--such as t, Z, and F--and randomization procedures. Randomization tests are based on the sample at hand. If I collect data on the income of males and females, the probability model and the basis for our estimates of error are repeated random allocations of the actual data values. I could compare observed differences across groups to a distribution based on this randomization. (We do this all the time in experiments, by the way, where the random sampling from a population model is rarely appropriate).

Now, it turns out that sample-theoretic tests are often good approximations of randomization tests. So, ultimately, I think tests from populations are useful and meaningful within this framework and can help to distinguish systematic from chance variation--just like with sample-based tests. The logic used to get there is a little different, but it doesn't have much affect on the practical meaning and use of tests. Of course, it might be better to just use randomization and permutation tests directly given they are easily available with all our modern computing power.

-
+1 for the sensible discussion; a few points though. Inferential machinery is unavailable for population analysis, but in many modeling cases, I'd question whether one ever has the population data to begin with -- often, it's not very hard to poke holes. So it's not always an appeal to a super population as the means to deploy inference. Rather than "super population", the better way is to assume a data generating process yielding, for example, the year to year test taking cohorts in question. That's where the stochastic component arises. –  ars Sep 14 '10 at 20:41
I don't think there as any disagreement here, except for the lack of inferential machinery for population analysis. Randomization tests are applicable to populations and can reasonably test whether the data generating process is likely due to a random generating process versus a systematic generating process. They do not assume random sampling and are a rather direct test of chance versus systematic variation. Our traditional tests happen to stand in pretty well for them. –  Brett Magill Sep 14 '10 at 22:45
That's true re: "lack of inferential machinery". Careless wording on my part, especially since I liked the point you made about randomization tests in your answer. –  ars Sep 14 '10 at 23:02
Thanks a lot for your answers. –  pbneau Sep 16 '10 at 18:52
sorry. I have difficulties to understand how i would compute permutations and what conclusions I'll be able to made for them. –  pbneau Sep 16 '10 at 18:53