Strategies for teaching the sampling distribution

Question

The tl;dr version What successful strategies do you employ to teach the sampling distribution (of a sample mean, for example) at an introductory undergraduate level?

The background

In September I'll be teaching an introductory statistics course for second-year social science (mainly political science and sociology) students using The Basic Practice of Statistics by David Moore. It'll be the fifth time I've taught this course and one issue I've consistently had is that the students have really struggled with the notion of the sampling distribution. It's covered as the background for inference and follows a basic introduction to probability with which they don't seem to have trouble after some initial hiccups (and by basic, I mean basic -- after all, many of these students have been self-selected into a specific course stream because they were trying to avoid anything with even a vague hint of "math"). I would guess that probably 60% leave the course with no to minimal understanding, about 25% understand the principle but not the connections to other concepts, and the remaining 15% fully understand.

The main issue

The trouble students seem to have is with the application. It's difficult to explain what the precise issue is other than to say they just don't get it. From a poll I conducted last semester and from exam responses, I think that part of the difficulty is confusion between two related and similar-sounding phrases (sampling distribution and sample distribution), so I don't use the phrase "sample distribution" anymore, but surely this is something that, while confusing at first, is easily grasped with a little effort and anyway it can't explain the general confusion of the concept of a sampling distribution.

(I realize that it might be me and my teaching that's at issue here! However, I think ignoring that uncomfortable possibility is reasonable to do since some students do seem to get it and overall everybody seems to do quite well...)

What I've tried

I had to argue with the undergraduate administrator in our department to introduce mandatory sessions in the computer lab thinking that repeated demonstrations might be helpful (before I started teaching this course there was no computing involved). While I think this helps overall understanding of the course material in general, I don't think it's helped with this specific topic.

One idea I've had is to simply not teach it at all or to not give it much weight, a position advocated by some (e.g. Andrew Gelman). I don't find this particularly satisfying since it has the whiff of teaching to the lowest common denominator and more importantly, denies strong and motivated students who want to learn more about statistical applications from really understanding how important concepts work (not only the sampling distribution!). On the other hand, the median student does seem to grasp p-values for example, so maybe they don't need to understand the sampling distribution anyway.

The question

What strategies do you employ to teach the sampling distribution? I know there are materials and discussions available (e.g. here and here and this paper which opens a PDF file) but I'm just wondering if I can get some concrete examples of what works for people (or I guess even what doesn't work so I'll know not to try it!). My plan now, as I plan my course for September, is to follow Gelman's advice and "deemphasize" the sampling distribution. I'll teach it, but I'll assure the students that this is a sort of FYI-only topic and will not appear on an exam (except perhaps as a bonus question?!). However, I'm really interested in hearing about other approaches people have used.

Answer 1

In my opinion, sampling distributions are the key idea of statistics 101. You might as well skip the course as skip that issue. However, I am very familiar with the fact that students just don't get it, seemingly no matter what you do. I have a series of strategies. These can take up a lot of time, but I recommend skipping / abbreviating other topics, so as to ensure that they get the idea of the sampling distribution. Here are some tips:

• Say it distinctly: I first explicitly mention that there 3 different distributions that we are concerned with: the population distribution, the sample distribution, and the sampling distribution. I say this over and over throughout the lesson, and then over and over throughout the course. Every time I say these terms I emphasize the distinctive ending: sam-ple, samp-ling. (Yes, students do get sick of this; they also get the concept.)
• Use pictures (figures): I have a set of standard figures that I use every time I talk about this. It has the three distributions pictured distinctly and typically labeled. (The labels that go with this figure are on the PowerPoint slide and include short descriptions, so they don't show up here, but obviously it's: the population at the top, then samples, then sampling distribution.)
  
• Give the students activities: The first time you introduce this concept, either bring in a roll of nickles (some quarters may disappear) or a bunch of 6-sided dice. Have the students form into small groups and generate a set of 10 values and average them. Then you can make a histogram on the board or with Excel.
• Use animations (simulations): I write some (comically inefficient) code in R to generate data & display it in action. This part is especially helpful when you transition to explaining the Central Limit Theorem. (Notice the Sys.sleep() statements, these pauses give me a moment to explain what is going on at each stage.)

N = 10
number_of_samples = 1000


iterations  = c(3, 7, number_of_samples)  
breakpoints = seq(10, 91, 3)  
meanVect    = vector()  
x           = seq(10, 90)  
height      = 30/dnorm(50, mean=50, sd=10)  
y           = height*dnorm(x, mean=50, sd=10)  
  
windows(height=7, width=5)  
par(mfrow=c(3,1), omi=c(0.5,0,0,0), mai=c(0.1, 0.1, 0.2, 0.1))  

for(i in 1:iterations[3]) {  
  plot(x,y, type="l", col="blue", axes=F, xlab="", ylab="")  
  segments(x0=20, y0=0, x1=20, y1=y[11], col="lightgray")  
  segments(x0=30, y0=0, x1=30, y1=y[21], col="gray")  
  segments(x0=40, y0=0, x1=40, y1=y[31], col="darkgray")  
  segments(x0=50, y0=0, x1=50, y1=y[41])  
  segments(x0=60, y0=0, x1=60, y1=y[51], col="darkgray")  
  segments(x0=70, y0=0, x1=70, y1=y[61], col="gray")  
  segments(x0=80, y0=0, x1=80, y1=y[71], col="lightgray")  
  abline(h=0)  
  
  if(i==1) {  
    Sys.sleep(2)  
  }  
  sample = rnorm(N, mean=50, sd=10)  
  points(x=sample, y=rep(1,N), col="green", pch="*")  
  
  if(i<=iterations[1]) {  
    Sys.sleep(2)  
  }  
  xhist1 = hist(sample, breaks=breakpoints, plot=F)  
  hist(sample, breaks=breakpoints, axes=F, col="green", xlim=c(10,90),  
       ylim=c(0,N), main="", xlab="", ylab="")  
  if(i==iterations[3]) {  
    abline(v=50)  
  }  
  
  if(i<=iterations[2]) {  
    Sys.sleep(2)  
  }  
  sampleMean = mean(sample)  
  segments(x0=sampleMean, y0=0, x1=sampleMean,   
           y1=max(xhist1$counts)+1, col="red", lwd=3)  
  
  if(i<=iterations[1]) {  
    Sys.sleep(2)  
  }  
  meanVect = c(meanVect, sampleMean)  
  hist(meanVect, breaks=x, axes=F, col="red", main="",   
       xlab="", ylab="", ylim=c(0,((N/3)+(0.2*i))))  
  if(i<=iterations[2]) {  
    Sys.sleep(2)  
  }  
}  
  
Sys.sleep(2)  
xhist2 = hist(meanVect, breaks=x, plot=F)  
xMean  = round(mean(meanVect), digits=3)  
xSD    = round(sd(meanVect), digits=3)  
histHeight = (max(xhist2$counts)/dnorm(xMean, mean=xMean, sd=xSD))  
lines(x=x, y=(histHeight*dnorm(x, mean=xMean, sd=xSD)),   
      col="yellow", lwd=2)  
abline(v=50)  
  
txt1 = paste("population mean = 50     sampling distribution mean = ",  
             xMean, sep="")  
txt2 = paste("SD = 10     10/sqrt(", N,") = 3.162     SE = ", xSD,  
            sep="")  
mtext(txt1, side=1, outer=T)  
mtext(txt2, side=1, line=1.5, outer=T)  

• Reinstantiate these concepts throughout the semester: I bring the idea of the sampling distribution up again each time we talk about the next subject (albeit typically only very briefly). The most important place for this is when you teach ANOVA, as the null hypothesis case there really is the situation in which you sampled from the same population distribution several times, and your set of group means really is an empirical sampling distribution. (For an example of this, see my answer here: How does the standard error work?.)

Answer 2

I have had some luck with reminding students that the sampling distribution is the distribution of the test statistic based on a random sample. I have students think about what would happen in the sampling process itself was biased - focusing on extreme cases. For example, what would the "sampling distribution" look like if our sampling process always picked the same (special) subset? Then I would consider what the "sampling distribution" would look like if our sampling process only picked two specific (special) subsets (each with probability 1/2). These are pretty simple to work out with the sample mean (especially for particular choices of "special" for the underlying population).

I think for some (clearly not all) students this seems to help them with the idea that the sampling distribution can be very different from the population distribution. I have also used the central limit theorem example that Michael Chernick mentioned with some success - especially with distributions that are clearly not normal (simulations really do seem to help).

Answer 3

I start back with the teaching of probability. I don't go into a lot of the formal definitions and rules (just not enough time), but show probability by simulation. The Monty Hall problem is a great example to use, I show through simulation (and then follow-up with the logic) that the strategy to switch gives a higher probability of winning. I point out that by simulation we were able to play the game many times (without risk or reward) to evaluate the strategies and that lets us choose the better strategy (if we are ever in that situation). Choosing the better strategy does not guarentee a win, but it gives us a better chance and helps choose between strategies. I then point out that how this will apply to the rest of the course is that it will help us choose strategies where there is a random component, but more realistic situations that we will be in.

Then when I introduce the sampling distribution I again start with simulation and say we want to develop strategies. Just like with the Monty Hall problem, in real life we will only be able to take 1 sample, but we can simulate a bunch of samples to help us develop a strategy. I then show simulations of many samples from the same population (known population in this case) and show the relationships that we learn from the simulations (histogram of the sample means), i.e. sample means clustered around true mean (mean of means is mean), smaller standard deviation of sampling distribution for bigger samples, more normal for bigger samples. The whole time I talk about repeating the ideas of simulation to choose strategies, just the same idea as the Monty Hall problem applied now to sample means instead of game shows. I then show the official rules and say that in addition to the simulations they can be proved mathematically, but I will not inflict the proofs on the entire class. I offer that if they really want to see the mathematical proofs they can come to an office hour and I will show them the math (nobody from the intro classes has taken me up on this yet).

Then when we get to inference I say that we will only be able to take 1 sample in the real world, just like we would only get to play the game 1 time (at most), but we can use the strategies we learned from simulating many samples to develop a strategy (z-test, t-test, or CI formula) that will give us the chosen properties (chance of being correct). Just like with the game, we don't know before we start if our final conclusion will be correct (and usually we still don't know afterwards), but we do know from the simulations and sampling distribution what the long term probability is using that strategy.

Do 100% of students have a perfect understanding? no, but I think more of them get the general idea that we can use simulation and math rules (that they are glad they don't have to look at, just trust the book/instructor) to choose a strategy/formula that has the desired properties.

Answer 4

This is a very important and well-thought out issue on your part. I do think the concept of the sampling distribution is very basic to understanding inference and definitely should be taught.

I have taught many introductory statistics courses, particularly in biostatistics. I teach the concept of sampling distribution and have approaches that I think are good but don't really have good feedback to determine how successful I have been with them. Anyway here is what I do.

First I try to give a simple definition. The sampling distribution is the distribution that the test statistic would have if the sampling process were repeated many times. It depends on the population distribution that the data are assumed to be generated from.

Although I think this is about as simple a definition as I can give I realize, it is not very simple and understanding of the concept will not come immediately in most cases. So follow this up with a basic example that reinforces what is said with the definition.

The example I would use is a sample of size $n$ that is independent and identically distributed as a normal distribution with mean $\mu$ and variance $\sigma^2$. The sample average, which is used as a point estimate or to form a test statistic for the population mean $\mu$, has a sampling distribution which is normal with mean $\mu$ and variance $\sigma^2$/n.

Then I would follow this up with an important application, the central limit theorem. In the simplest terms, the central limit theorem says that for many distributions that are not normal, the sampling distribution for the sample mean will be close to a normal distribution when the sample size $n$ is large. To illustrate this take distributions like the uniform (a bimodal distribution would also be good to look at) and show what the sampling distribution for the mean looks like for sample sizes of 3, 4, 5, 10 and 100. The student can see how the shape of the distribution changes from something that does not look normal at all for small $n$ to something that looks very much like a normal distribution for large $n$.

To convince the student that these sampling distributions really do have these shapes have the students conduct simulations generating many samples of various sizes and compute the sample means. Then have them generate histograms for these estimates of the mean. I would also suggest applying a physical demonstration showing how this works using a quincunx board. While doing this you point out how the device generates samples of the sum of independent Bernoulli trials where the probability of going left or right at each level equals 1/2. The resulting stacks at the bottom represent a histogram for this sampling distribution (the binomial) and its shape can be seen to look approximately normal after a large number of balls land at the bottom of the quincunx, a demonstration of the DeMoivre-Laplace version of the central limit theorem through sampling distributions.

Answer 5

I think it would be good to put a 'population' of numbers in a bag ( ranging for example from 1-10). You could make your own tiles, or use coins, playing cards etc.

Get students to sit in groups (5 or more) and each picks a number out of the bag. Each group then calculates the mean value for their group. Tell them that earlier you worked out the population mean, plot it on a histogram and get a member of each group to come and plot their sample mean on a histogram around this. Get them to do this exercise a few times to 'build up the histogram'.

You will then be able to graphically show the variation in sample means around the population mean. Work out the variation in sample means compared to the population mean. I think students distinctly remember doing such a practical exercise and the concept of sampling variation will come back to them more easily as a result. It might sound a bit babyish but students sometimes just like a chance to do something active....there aren't many opportunities to do this in statistics.