9
$\begingroup$

Given the fact the finite population correction effectively reduces the standard error of the sampling distribution, which (compared with the lack of a finite population correction) will increase the test statistic from a hypothesis test (and therefore increase the probability that the null will be rejected), it seems quite a powerful tool to use without a strong justification.

Here is one scenario I feel it is justified to use it:

[Edit 2 - I have changed the scenario here slightly following a very valid point from Steve. I don't want non-response rates to distract from the focus of the finite population correction].

I have a company which has new leadership in the past 12 months. During the old leadership, I performed a random sample on 50% of the employees (with a 100% response rate). Under new leadership, I conducted the same survey on a random sample of 45% of the employees (again, with a 100% response rate)

If I want to see how the results to some questions in the survey have changed in a statistically significant way, I should apply the finite population correction. I am comparing two sets of employees at two specific points in time. I don't care about anyone other than the population working within the company at those two points in time.

[Edit 2 - To clarify my concern]

In addition, those who were not working in the firm at those two points in time would not be able to answer the survey anyway. Taking a basic question like 'I have enjoyed working here in the last couple of months,' only those who have actually worked at the firm could reasonably answer. In which case, assuming an infinite population seems like it is applying an arbitrary restriction on the sampling distribution.

Here is one scenario I feel it wouldn't be justified to use it:

I carry out a questionnaire to a sample of customers who bought products from me to assess their interests, in order for me to inform me how best to diversify my business for the next couple of years. While this sample is 20% of all those customers I have served since I opened, I could perceive the population here to be inclusive of all those customers who haven't bought products from me (because they aren't in the area or my prices were too high or they didn't know about my shop etc.) and therefore I have a population which leaves my sample to be <1% of it in size).

Questions:

Do people agree with these my interpretation of the two example scenarios?

When producing a piece of work, is it good practice to include the justification why you are using a finite population correction?

Is this largely a assessment of 'who' the population is and whether you want your results to apply just to the 'known' population or whether you are seeking to provide something that has a wider level of application than just the 'known' population?

$\endgroup$

2 Answers 2

8
$\begingroup$

You are correct about the second scenario, for the reason you give, but not about the first scenario. The theory of the finite population correction (fpc) applies only to a random sample without replacement (Lohr (2009) Sec 2.8,pp 51-530. The key word is random. The hallmark of a random sample is that selection is determined by random numbers or the physical equivalent. In your first scenario the 45% of the population who responded were not selected by random numbers. The same would be true if the 45% were part of an even large random sample of the population: response is not governed by random numbers.

Even if you have a sample of a substantial part of the population with (near) 100% response, you should still omit the fpc if the purpose of your study is to develop predictions, estimate odds ratios, or to otherwise test hypotheses or quote p-values. The reasoning is interesting (Cochran, 1977, p.39): For a finite population it is seldom of scientific interest to ask if a null hypothesis (e.g. that two proportions are equal) is exactly true. Except by a very rare chance, it will not be, as one would discover this by enumerating the entire population. This leads to the adoption of a "superpopulation" viewpoint, which is taken by almost all statisticians these days. Your second scenario is a variant of this. See also Deming(1966) pp 247-261 "Distinction between enumerative and analystic studies"; Korn and Graubard (1999), p. 227.

ADDED NOV 26 I should have noted that the finite population correction is a minor concern here. The major problem is the 55% non-response and the subsequent non-response bias. Survey professionals universally agree that it is better to take a smaller manageable sample and to focus on reducing non-response by personalizing the initial contacts and by following-up with non-responders. Post-survey weighting fixes may also help, but will increase standard errors.

In summary, to answer your three questions:

  1. Your interpretation of the first scenario is incorrect.
  2. You really don't need to say anything. If your goal is to describe only the finite population from which you drew the sample, then you can mention that you omit the fpc because the effect is miniscule. Otherwise, when you do hypothesis testing or prediction, you could mention that omit the fpc, but I've never seen anyone do it.
  3. The decision of whether to use the fpc is the assessment you describe in the question. So the answer is "Yes".

Additional discussion See a related CV discussion here.

References

Cochran, W. G. (1977). Sampling techniques (3rd Ed.). New York: Wiley.

Deming, W. E. (1966). Some theory of sampling. New York: Dover Publications.

Korn, E. L., & Graubard, B. I. (1999). Analysis of health surveys (Wiley series in probability and statistics). New York: Wiley.

Levy, Paul S, and Stanley Lemeshow. 2008. Sampling of populations : methods and applications. Wiley series in survey methodology. Hoboken, N.J: Wiley.

Lohr, Sharon L. 2009. Sampling: Design and Analysis. Boston, MA: Cengage Brooks/Cole.

$\endgroup$
8
  • $\begingroup$ Thanks for the reply Steve. Good point re: first scenario. If we assume it is a random sample (i.e. instead of 45% responding, I approached 45% randomly, with a 100% response rate), do you feel the reasoning by Cochran still applies? I am struggling to see the benefit in assuming a ‘super population’ for the first scenario (...). $\endgroup$
    – NickB2014
    Nov 26, 2014 at 13:57
  • 1
    $\begingroup$ (...) There is no one outside of the firm (except maybe those who have left recently but let’s assume no one has quit, for simplicity) who can answer a question such as ‘I have enjoyed working for the firm in the past 12 months’ so why am I being forced to consider a sampling distribution that is inclusive of them? To turn Cochran’s quote on its head, it feels that, in this scenario, there is limited use in testing the null against the ‘super population.’ $\endgroup$
    – NickB2014
    Nov 26, 2014 at 13:58
  • 1
    $\begingroup$ I added explicit answers. But, yes, the reasoning of Cochran applies for any finite population analysis that is not descriptive. If you wanted to do a hypothesis test for comparing two groups, for example, then you have a null hypothesis of exact equality of, say, means. That can't be true in a finite population; so you must mean it to apply in other circumstances, left vague, of which the current population is an instance. It could be future observations. $\endgroup$ Nov 27, 2014 at 0:22
  • 1
    $\begingroup$ Cochran's reasoning does apply when you want your analysis to say something that will apply outside the original finite population at the time the sample was taken. Your prediction is of that sort; so is any kind of theoretical modeling. There is are instances when you want to do a hypothesis test about finite populations (Levy and Lemeshow, 2008, 429). In that example, a population will get a vaccination campaign if vaccination coverage is not high enough. The formal language of hypothesis testing is used, but it's really a decision problem about the population. $\endgroup$ Nov 27, 2014 at 0:33
  • 1
    $\begingroup$ There is are instances when you want to do a hypothesis test about a finite population (Levy and Lemeshow, 2008, 429). In that example, a population will get a vaccination campaign if vaccination coverage is not high enough. The sample is taken to estimate the coverage. The formal language of hypothesis testing is used, but it's really a decision problem. $\endgroup$ Nov 27, 2014 at 0:39
0
$\begingroup$

I agree with the accepted answer that it would not be appropriate to utilize the finite population correction. However, I think Rubin's potential outcome framework explains why this would be inappropriate much, much more persuasively than Cochran.

Presumably, by comparing, say worker satisfaction under the old boss and the new boss, the intent is to answer a causal question. What's the average treatment effect of replacing the old boss with the new boss? Rubin says that the best way to answer a question like that is to think about it as a randomized experiment. At a specific point in time, $a_1$, there are two possibilities and two corresponding potential outcomes for each person (unit). One possibility is that the old boss is retained. Then satisfaction is measured at point $a_2$. For each unit, denote this measure as $c_i$. The other possibility is the new boss is hired. Then satisfaction is measured at point $t_2$. Denote this measure as $t_i$.

We want to compare

$$ \frac{1}{N} \sum^N_{i=1} t_i - \frac{1}{N} \sum^N_{i=1} c_i $$

The problem is that even in a randomized experiment, we can't do that because for each unit, we always only observe either $t_i$ or $c_i$. Thus, questions of causal inference, and this extends beyond randomized experiments, are really questions about missing data.

In a randomized experiment, we have guarantees that difference between the sample mean for the treatment and control group will give us an unbiased estimate of the average treatment effect.

In your example, we're missing the whole $c_i$ vector, so it's more difficult. The key point here though is that even if you every one responds to your survey, you're missing half of the data. There are ways to try to sort of get around this (propensity scores, etc.) so here we'll just assume that you do sort of get around it.

Now you might think we can just go ahead and apply the finite population correction to our estimate of the variance of the difference in means as long as we account for the fact that the real population is $N = N_t + N_c$. However, returning to our grounding example of a randomized experiment, we see that assignment is random but not independent (assuming fixed sizes for the treatment and control group). The last unit assigned can be perfectly predicted by the assignment of the previous units.

That means we have a non-zero covariance term. Letting $W_i$ represent a binary for inclusion (1, 0), we have

$$ Var(\frac{1}{N_t} \sum^N_{i=1} W_i t_i - \frac{1}{N_c} \sum^N_{i=1} (1 - W_i) c_i) = \\ Var(\frac{1}{N_t} \sum^N_{i=1} W_i t_i) + Var(\frac{1}{N_c} \sum^N_{i=1} (1 - W_i) c_i) - 2 \cdot Cov(\frac{1}{N_t} \sum^N_{i=1} W_i t_i, \frac{1}{N_c} \sum^N_{i=1} (1 - W_i)c_i) $$

The problem is that because we never observe $t_i$ and $c_i$ together, it turns out, we can't estimate the covariance. That forces us to take a conservative approach that approximates dropping the finite population correction.

For more details see my response here.

For a good introduction to this perspective, see Causal Inference by Rubin and Imbens. A recent article by Abadie, Athey, Imbens, Wooldridge that exploits this perspective, "Sampling-based vs. Design-based Uncertainty in Regression Analysis", recently appeared in Econometrica. Here's a link to the pre-print.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.