Statistical inference when the sample "is" the population

Question

Imagine you have to do reporting on the numbers of candidates who yearly take a given test. It seems rather difficult to infer the observed % of success, for instance, on a wider population due to the specifity of the target population. So you may consider that these data represent the whole population.

Are results of tests indicating that the proportions of males and females are different really correct? Does a test comparing observed and theoretical proportions appear to be a correct one, since you consider a whole population (and not a sample)?

Answer 1

There may be varying opinions on this, but I would treat the population data as a sample and assume a hypothetical population, then make inferences in the usual way. One way to think about this is that there is an underlying data generating process responsible for the collected data, the "population" distribution.

In your particular case, this might make even more sense since you will have cohorts in the future. Then your population is really cohorts who take the test even in the future. In this way, you could account for time based variations if you have data for more than a year, or try to account for latent factors through your error model. In short, you can develop richer models with greater explanatory power.

Answer 2

Actually, if you're really positive you have the whole population, there's even no need to go into statistics. Then you know exactly how big the difference is, and there is no reason whatsoever to test it any more. A classical mistake is using statistical significance as "relevant" significance. If you sampled the population, the difference is what it is.

On the other hand, if you reformulate your hypothesis, then the candidates can be seen as a sample of possible candidates, which would allow for statistical testing. In this case, you'd test in general whether male and female differ on the test at hand.

As ars said, you can use tests of multiple years and add time as a random factor. But if your interest really is in the differences between these candidates on this particular test, you cannot use the generalization and testing is senseless.

Answer 3

Traditionally, statistical inference is taught in the context of probability samples and the nature of sampling error. This model is the basis for the test of significance. However, there are other ways to model systematic departures from chance and it turns out that our parametric (sampling based) tests tend to be good approximations of these alternatives.

Parametric tests of hypotheses rely on sampling theory to produce estimates of likely error. If a sample of a given size is taken from a population, knowledge of the systematic nature of sampling makes testing and confidence intervals meaningful. With a population, sampling theory is simply not relevant and tests are not meaningful in the traditional sense. Inference is useless, there is nothing to infer to, there is just the thing...the parameter itself.

Some get around this by appealing to super-populations that the current census represents. I find these appeals unconvincing--parametric tests are premised on probability sampling and its characteristics. A population at a given time may be a sample of a larger population over time and place. However, I don't see any way that one could legitimately argue that this is a random (or more generally any form form of a probability) sample. Without a probability sample, sampling theory and the traditional logic of testing simply do not apply. You may just as well test on the basis of a convenience sample.

Clearly, to accept testing when using a population, we need to dispense with the basis of those tests in sampling procedures. One way to do this is to recognize the close connection between our sample-theoretic tests--such as t, Z, and F--and randomization procedures. Randomization tests are based on the sample at hand. If I collect data on the income of males and females, the probability model and the basis for our estimates of error are repeated random allocations of the actual data values. I could compare observed differences across groups to a distribution based on this randomization. (We do this all the time in experiments, by the way, where the random sampling from a population model is rarely appropriate).

Now, it turns out that sample-theoretic tests are often good approximations of randomization tests. So, ultimately, I think tests from populations are useful and meaningful within this framework and can help to distinguish systematic from chance variation--just like with sample-based tests. The logic used to get there is a little different, but it doesn't have much affect on the practical meaning and use of tests. Of course, it might be better to just use randomization and permutation tests directly given they are easily available with all our modern computing power.

Answer 4

Assume the results indicate that candidates differ along lines of gender. For example, the proportion of those who completed the tests are as follows: 40% female and 60% male. To suggest the obvious, 40% is different than 60%. Now what is important is to decide: 1) your population of interest; 2) how your observations relate to the population of interest. Here are some details about these two issues:

1. If your population of interest is just the candidates you have observed (e.g., the 100 candidates who applied to a university in 2016), you do not need to report statistical significance tests. This is because your population of interest was completely sampled...all you care about is the 100 candidates on which you have complete data. That is, 60% is, full stop, different than 40%. The kind of question this answers is, were there gender differences in the population of 100 that applied to the program? This is a descriptive question and the answer is yes.

2. However, many important questions are about what will happen in different settings. That is, many researchers want to come up with trends about the past that help us predict (and then plan for) the future. An example question in this regard would be, How likely are future tests of candidates likely to be different along lines of gender? The population of interest is then broader than in scenario #1 above. At this point, an important question to ask is: Is your observed data likely to be representative of future trends? This is an inferential question, and based on the info provided by the original poster, the answer is: we don't know.

In sum, what statistics you report depend on the type of question you want to answer.

Thinking about basic research design may be most helpful (try here: http://www.socialresearchmethods.net/kb/design.php). Thinking about superpopulations may be of help if you want more advanced info (here is an article that may help: http://projecteuclid.org/euclid.ss/1023798999#ui-tabs-1).

Answer 5

If you consider whatever it is that you are measuring to be a random process, then yes statistical tests are relevant. Take for example, flipping a coin 10 times to see if it is fair. You get 6 heads and 4 tails - what do you conclude?