Statistical inference when the sample “is” the population

Question

Imagine you have to do reporting on the numbers of candidates who yearly take a given test. It seems rather difficult to infer the observed % of success, for instance, on a wider population due to the specifity of the target population. So you may consider that these data represent the whole population.

Are results of tests indicating that the proportions of males and females are different really correct? Does a test comparing observed and theoretical proportions appear to be a correct one, since you consider a whole population (and not a sample)?

Answer 1

There may be varying opinions on this, but I would treat the population data as a sample and assume a hypothetical population, then make inferences in the usual way. One way to think about this is that there is an underlying data generating process responsible for the collected data, the "population" distribution.

In your particular case, this might make even more sense since you will have cohorts in the future. Then your population is really cohorts who take the test even in the future. In this way, you could account for time based variations if you have data for more than a year, or try to account for latent factors through your error model. In short, you can develop richer models with greater explanatory power.

Answer 2

Actually, if you're really positive you have the whole population, there's even no need to go into statistics. Then you know exactly how big the difference is, and there is no reason whatsoever to test it any more. A classical mistake is using statistical significance as "relevant" significance. If you sampled the population, the difference is what it is.

On the other hand, if you reformulate your hypothesis, then the candidates can be seen as a sample of possible candidates, which would allow for statistical testing. In this case, you'd test in general whether male and female differ on the test at hand.

As ars said, you can use tests of multiple years and add time as a random factor. But if your interest really is in the differences between these candidates on this particular test, you cannot use the generalization and testing is senseless.

Answer 3

Traditionally, statistical inference is taught in the context of probability samples and the nature of sampling error. This model is the basis for the test of significance. However, there are other ways to model systematic departures from chance and it turns out that our parametric (sampling based) tests tend to be good approximations of these alternatives.

Parametric tests of hypotheses rely on sampling theory to produce estimates of likely error. If a sample of a given size is taken from a population, knowledge of the systematic nature of sampling makes testing and confidence intervals meaningful. With a population, sampling theory is simply not relevant and tests are not meaningful in the traditional sense. Inference is useless, there is nothing to infer to, there is just the thing...the parameter itself.

Some get around this by appealing to super-populations that the current census represents. I find these appeals unconvincing--parametric tests are premised on probability sampling and its characteristics. A population at a given time may be a sample of a larger population over time and place. However, I don't see any way that one could legitimately argue that this is a random (or more generally any form form of a probability) sample. Without a probability sample, sampling theory and the traditional logic of testing simply do not apply. You may just as well test on the basis of a convenience sample.

Clearly, to accept testing when using a population, we need to dispense with the basis of those tests in sampling procedures. One way to do this is to recognize the close connection between our sample-theoretic tests--such as t, Z, and F--and randomization procedures. Randomization tests are based on the sample at hand. If I collect data on the income of males and females, the probability model and the basis for our estimates of error are repeated random allocations of the actual data values. I could compare observed differences across groups to a distribution based on this randomization. (We do this all the time in experiments, by the way, where the random sampling from a population model is rarely appropriate).

Now, it turns out that sample-theoretic tests are often good approximations of randomization tests. So, ultimately, I think tests from populations are useful and meaningful within this framework and can help to distinguish systematic from chance variation--just like with sample-based tests. The logic used to get there is a little different, but it doesn't have much affect on the practical meaning and use of tests. Of course, it might be better to just use randomization and permutation tests directly given they are easily available with all our modern computing power.

Answer 4

If you consider whatever it is that you are measuring to be a random process, then yes statistical tests are relevant. Take for example, flipping a coin 10 times to see if it is fair. You get 6 heads and 4 tails - what do you conclude?