6
$\begingroup$

At academic institutions dedicated to teaching we often use our current students taking a specific class, the accessible population, as a random sample when our actual sample includes all students who will take this class in the future. Has anyone ever studied how reasonable this is? Thank you.

$\endgroup$
6
$\begingroup$

This is an important question, made explicit by Deming and Stephan (1941), who used first the word "superpopulation" to describe the approach with that name: assume that the current population is itself a sample from a larger, hypothetical, population. The concept is implicit also in Cochran (1939). See Stanek, 2000b, where I first found the reference to Cochran's paper.

If the students each year are drawn from the this superpopulation and teaching remains the same, then consider the available population to be a simple random sample and use the appropriate survey-design-based analyses (Deming, 1966, pp 247-261). There are also model-based superpopulation solutions, e.g. that observations are drawn from normal distributions, but these are stronger assumptions. I would also avoid inference based on likelihood ratios.

If, however, there are random or systematic (e.g. temporal trend) differences between students each year, then you would need several years of data to estimate these effects and incorporate them into into your analyses.

If the teaching content (or instructor) also changes from year-to-year, then you have an additional source of difference that will be difficult to predict.

The bottom line: you can analyze the class as if it represents future classes, but you must qualify your conclusions by stating the problems with this assumption.

I have answered related questions elsewhere on SO. See, e.g.

Applying inferential statistics for census data

Justifying the use of finite population correction

Adjusting any power analysis with FPC?

For some other references on the superpopulation approach, see: Korn and Graubard, 1999,p.227); Gelman, 2009; and a couple of unpublished notes by Ed Stanek (2000 a,b). The first paper contains an incomplete set of references.

References

Cochran, W. G. (1939). "The use of analysis of variance in enumeration by sampling.,"Journal of the American Statistical Association,34:492-51

Cochran, W. G. (1977). Sampling techniques (3rd Ed.). New York: Wiley.

Deming, W Edwards, and Frederick F Stephan. (1941). On the interpretation of censuses as samples. Journal of the American Statistical Association 36, no. 213: 45-49

Deming, W. E. (1966). Some theory of sampling. New York: Dover Publications.

Andrew Gelman, 2009. How does statistical analysis differ when analyzing the entire population rather than a sample? http://andrewgelman.com/2009/07/03/how_does_statis/

Korn, E. L., & Graubard, B. I. (1999). Analysis of health surveys (Wiley series in probability and statistics). New York: Wiley.

Ed Stanek (2000a) Ideas on Superpopulation Models and Inference http://www.umass.edu/cluster/ed/unpublication/yr2000/c00ed62.PDF

Ed Stanek (2000b) Superpopulations and Superpopulation Models http://www.umass.edu/cluster/ed/unpublication/yr2000/c00ed64v1.PDF

| cite | improve this answer | |
$\endgroup$
9
$\begingroup$

At its face value, this is a convenience sample. Real sampling involves randomization, and I don't think any university will allow you to randomly put students to sections. There's undoubtedly an issue of self-selection that produces biased samples with skewed prevalences of students with different backgrounds and characteristics. Only the more responsible students will take MWF 8:00am classes, and those who need to work during the day may prefer late night classes, etc.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ How about if we use all the sections of a class, say elementary statistics. Is that a random sample of all students who will take statistics at the institution? $\endgroup$ – Bill Abrams May 7 '15 at 16:58
  • $\begingroup$ If you use all sections of a class, you have a census of the finite population of the class. The population of students who will take a class is a hypothetical one, and I will be uncomfortable talking about a sample of unicorns. $\endgroup$ – StasK May 8 '15 at 20:24
  • 1
    $\begingroup$ As a continuation of @StasK's comment, the distribution of class times and capacity may change over time changing the distribution. However, what I think is the more important question, is not how biased the sample is, but how does that bias affect your decisions, they may very well be largely invariant to the sampling mechanism (e.g. the model based approach). $\endgroup$ – Jonathan Lisic May 11 '15 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.