Why do we adjust for within-group variability in statistical testing of differences in group means?

Question

I hope this question fits the forum. I'm not a statistician, but have received decent training in statistics and use statistical modeling in my daily work. I've been tasked to give a very condensed "info-package-plus-practice-workshop" about how to run and interpret statistical analyses in social sciences to a group of researchers who have little to no statistical training or experience.

There will be a lot of practical exercises but first I'd like to say something about why we use statistical analyses in the first place. I was thinking of going about it by focusing on a design with two groups and a continuous outcome/response variable and presenting and answering the question of "why can't we just look at the raw group means and see which group has a higher mean?"

I was then going to explain that if we just look at the raw group difference, we ignore within-group variability, and statistical tests help us with that by adjusting the between-group difference to within-group variability¹.

However, I realized I can't quite find the words to express, in plain, common sense language and/or through a practical example, WHY large within-group variability undermines between-group differences. i.e. why do we need to adjust for within-group variability. If we have a large between-group difference, why can't we just go by that and ignore the within-group variability?

I'd be grateful if someone could help me find the right words (or tell me this is not a good way to teach about this).

¹I will mention other reasons for conducting statistical tests too, but I already have an idea how to explain those.

Answer 1

The issue is that you will always have differences between the responses observed in your two groups, simply because of random variations. Importantly, you will have such differences even if your groups do not differ systematically at all - e.g., if you assigned participants to the two groups completely at random.

The idea of statistical testing it to observe these differences in sample means and try to infer whether these are indeed due to chance, or whether we can conclude that the groups do indeed differ on their true unobservable mean. And for that, there is simply no way around comparing the difference in observed means to the variability in the data. Specifically, since we are capturing the between-groups variability by calculating the two separate group means, we are left with the within-group variability. And for instance, dividing the difference in observed means by the within-group variability is rather precisely what a t-test for group differences does.

As an illustration, suppose we run two experiments, with two groups each. Results are as below. The differences between group means are about the same size of 1 each - but in one experiment, we can be much more confident that this observed difference is indeed due to an underlying difference in true unobservable group means, while in the other experiment, it may simply be due to sampling noise. And the difference lies precisely in the within-group variability.

R code:

nn <- 20
set.seed(1)
dataset_1 <- data.frame(A=rnorm(nn,0,.1),B=rnorm(nn,1,.1))
set.seed(1)
dataset_2 <- data.frame(A=rnorm(nn,0,1),B=rnorm(nn,1,1))

par(mfrow=c(1,2),las=1)
plot(c(0.5,2.5),range(dataset_1),type="n",xaxt="n",xlab="Group",ylab="Response",
    main="Low intra-group variation")
axis(1,at=c(1,2),labels=c("A","B"))
points(runif(nn,0.8,1.2),dataset_1$A,pch=19)
lines(c(0.8,1.2),rep(mean(dataset_1$A),2),lwd=2)
points(runif(nn,1.8,2.2),dataset_1$B,pch=19)
lines(c(1.8,2.2),rep(mean(dataset_1$B),2),lwd=2)
#
plot(c(0.5,2.5),range(dataset_2),type="n",xaxt="n",xlab="Group",ylab="Response",
    main="High intra-group variation")
axis(1,at=c(1,2),labels=c("A","B"))
points(runif(nn,0.8,1.2),dataset_2$A,pch=19)
lines(c(0.8,1.2),rep(mean(dataset_2$A),2),lwd=2)
points(runif(nn,1.8,2.2),dataset_2$B,pch=19)
lines(c(1.8,2.2),rep(mean(dataset_2$B),2),lwd=2)

Answer 2

Stephen gives a very good answer (+1). I wrote an entirely non-mathematical answer on Medium. Why analyze variances to compare means

which may appeal to the less mathematical among your students. (Having tutored and consulted with students in the social sciences, I am sure there are some who hate math).