45
$\begingroup$

Imagine you have to do reporting on the numbers of candidates who yearly take a given test. It seems rather difficult to infer the observed % of success, for instance, on a wider population due to the specifity of the target population. So you may consider that these data represent the whole population.

Are results of tests indicating that the proportions of males and females are different really correct? Does a test comparing observed and theoretical proportions appear to be a correct one, since you consider a whole population (and not a sample)?

$\endgroup$
31
$\begingroup$

There may be varying opinions on this, but I would treat the population data as a sample and assume a hypothetical population, then make inferences in the usual way. One way to think about this is that there is an underlying data generating process responsible for the collected data, the "population" distribution.

In your particular case, this might make even more sense since you will have cohorts in the future. Then your population is really cohorts who take the test even in the future. In this way, you could account for time based variations if you have data for more than a year, or try to account for latent factors through your error model. In short, you can develop richer models with greater explanatory power.

$\endgroup$
  • 4
    $\begingroup$ Just came across this post from A Gelman, How does statistical analysis differ when analyzing the entire population rather than a sample?, j.mp/cZ1WSI. A good starting point about diverging opinions on the concept of a "super-population". $\endgroup$ – chl Oct 31 '10 at 17:20
  • 2
    $\begingroup$ @chl: interesting -- reminds me that Gelman had a discussion of finite/super population inference being comparable to fixed-/random-effects in his paper on ANOVA [ stat.columbia.edu/~gelman/research/published/econanova3.pdf ]. $\endgroup$ – ars Oct 31 '10 at 17:53
  • $\begingroup$ +1 I just came back to this again (through google). I think that your answer is spot on. $\endgroup$ – Shane Oct 31 '10 at 23:37
24
$\begingroup$

Actually, if you're really positive you have the whole population, there's even no need to go into statistics. Then you know exactly how big the difference is, and there is no reason whatsoever to test it any more. A classical mistake is using statistical significance as "relevant" significance. If you sampled the population, the difference is what it is.

On the other hand, if you reformulate your hypothesis, then the candidates can be seen as a sample of possible candidates, which would allow for statistical testing. In this case, you'd test in general whether male and female differ on the test at hand.

As ars said, you can use tests of multiple years and add time as a random factor. But if your interest really is in the differences between these candidates on this particular test, you cannot use the generalization and testing is senseless.

$\endgroup$
15
$\begingroup$

Traditionally, statistical inference is taught in the context of probability samples and the nature of sampling error. This model is the basis for the test of significance. However, there are other ways to model systematic departures from chance and it turns out that our parametric (sampling based) tests tend to be good approximations of these alternatives.

Parametric tests of hypotheses rely on sampling theory to produce estimates of likely error. If a sample of a given size is taken from a population, knowledge of the systematic nature of sampling makes testing and confidence intervals meaningful. With a population, sampling theory is simply not relevant and tests are not meaningful in the traditional sense. Inference is useless, there is nothing to infer to, there is just the thing...the parameter itself.

Some get around this by appealing to super-populations that the current census represents. I find these appeals unconvincing--parametric tests are premised on probability sampling and its characteristics. A population at a given time may be a sample of a larger population over time and place. However, I don't see any way that one could legitimately argue that this is a random (or more generally any form form of a probability) sample. Without a probability sample, sampling theory and the traditional logic of testing simply do not apply. You may just as well test on the basis of a convenience sample.

Clearly, to accept testing when using a population, we need to dispense with the basis of those tests in sampling procedures. One way to do this is to recognize the close connection between our sample-theoretic tests--such as t, Z, and F--and randomization procedures. Randomization tests are based on the sample at hand. If I collect data on the income of males and females, the probability model and the basis for our estimates of error are repeated random allocations of the actual data values. I could compare observed differences across groups to a distribution based on this randomization. (We do this all the time in experiments, by the way, where the random sampling from a population model is rarely appropriate).

Now, it turns out that sample-theoretic tests are often good approximations of randomization tests. So, ultimately, I think tests from populations are useful and meaningful within this framework and can help to distinguish systematic from chance variation--just like with sample-based tests. The logic used to get there is a little different, but it doesn't have much affect on the practical meaning and use of tests. Of course, it might be better to just use randomization and permutation tests directly given they are easily available with all our modern computing power.

$\endgroup$
  • 2
    $\begingroup$ +1 for the sensible discussion; a few points though. Inferential machinery is unavailable for population analysis, but in many modeling cases, I'd question whether one ever has the population data to begin with -- often, it's not very hard to poke holes. So it's not always an appeal to a super population as the means to deploy inference. Rather than "super population", the better way is to assume a data generating process yielding, for example, the year to year test taking cohorts in question. That's where the stochastic component arises. $\endgroup$ – ars Sep 14 '10 at 20:41
  • 2
    $\begingroup$ I don't think there as any disagreement here, except for the lack of inferential machinery for population analysis. Randomization tests are applicable to populations and can reasonably test whether the data generating process is likely due to a random generating process versus a systematic generating process. They do not assume random sampling and are a rather direct test of chance versus systematic variation. Our traditional tests happen to stand in pretty well for them. $\endgroup$ – Brett Sep 14 '10 at 22:45
  • $\begingroup$ That's true re: "lack of inferential machinery". Careless wording on my part, especially since I liked the point you made about randomization tests in your answer. $\endgroup$ – ars Sep 14 '10 at 23:02
  • $\begingroup$ sorry. I have difficulties to understand how i would compute permutations and what conclusions I'll be able to made for them. $\endgroup$ – pbneau Sep 16 '10 at 18:53
  • $\begingroup$ Is bootstrapping not a valid alternative? How does bootstrapping fail to resolve the need to make either of these assumptions? $\endgroup$ – Chernoff Jul 12 '17 at 15:40
3
$\begingroup$

Assume the results indicate that candidates differ along lines of gender. For example, the proportion of those who completed the tests are as follows: 40% female and 60% male. To suggest the obvious, 40% is different than 60%. Now what is important is to decide: 1) your population of interest; 2) how your observations relate to the population of interest. Here are some details about these two issues:

  1. If you population of interest is just the candidates you have observed (e.g., the 100 candidates who applied to a university in 2016), you do not need to report statistical significance tests. This is because your population of interest was completely sampled...all you care about is the 100 candidates on which you have complete data. That is, 60% is, full stop, different than 40%. The kind of question this answers is, were there gender differences in the population of 100 that applied to the program? This is a descriptive question and the answer is yes.

  2. However, many important questions are about what will happen in different settings. That is, many researchers want to come up with trends about the past that help us predict (and then plan for) the future. An example question in this regard would be, How likely are future tests of candidates likely to be different along lines of gender? The population of interest is then broader than in scenario #1 above. At this point, an important question to ask is: Is your observed data likely to be representative of future trends? This is an inferential question, and based on the info provided by the original poster, the answer is: we don't know.

In sum, what statistics you report depend on the type of question you want to answer.

Thinking about basic research design may be most helpful (try here: http://www.socialresearchmethods.net/kb/design.php). Thinking about superpopulations may be of help if you want more advanced info (here is an article that may help: http://projecteuclid.org/euclid.ss/1023798999#ui-tabs-1).

$\endgroup$
2
$\begingroup$

If you consider whatever it is that you are measuring to be a random process, then yes statistical tests are relevant. Take for example, flipping a coin 10 times to see if it is fair. You get 6 heads and 4 tails - what do you conclude?

$\endgroup$
  • 1
    $\begingroup$ I don't really understand how the conclusion you reached about the issue of tossing a coin is related to the question asked. Maybe you could expand a little bit on that point? Statistical tests seem to be relevant to the extent they help to infer observed results to a larger population, whether it be a reference or general population. The question here seems to be: Given that the sample is close to the population of test takers for a fixed period of time (here, one year), is classical inference the right way to reach a decision about possible differences at the individual level? $\endgroup$ – chl Sep 14 '10 at 19:43
  • 1
    $\begingroup$ @chl Yes, but it seems to OP is trying to infer an underlying probability of success. The tests compare the observed proportions to the theoretical distribution to determine if there is a difference for a given level of confidence. You are testing for any form of randomness, not just sampling error randomness. $\endgroup$ – James Sep 15 '10 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.