Comparing means across sub-groups?

Question

Bit of a stats newbie question here. I've got about 10,000 rows of data about pupil performance, annotated with school and teacher details, that look as follows:

Pupil      School     Grade(1-5)    Teacher
123        Morley     4             Adams
124        Westlake   3             Morris
125        Morley     5             Adams
126        Westlake   2             Philips
127        St Xavier  4             Smith

I've calculated that the overall mean grade is 3.4 with standard deviation 1.0.

Now, say I want to work out whether one school (or one teacher) has overall a significantly lower or higher performance than average.

If I calculate that the overall mean grade at Morley is 3.9, with s.d. 0.8, the overall mean grade at Westlake is 3.2, with s.d. 0.7, and the overall mean at St Xavier is 3.8, with s.d. 0.5, what test can I use to say whether any of these schools have a significantly different performance from the average?

From my knowledge so far, I'm guessing I want to use some form of t-test, probably Student's t-test - although I'm not sure how to adapt it for multiple groups.

(Of course, there are lots of factors influencing why grades might be lower at one school than another and yet that school might still be performing well - I'm not getting that deep here. I just want to know how to compare means across sub-groups.)

Many thanks for your help.

Answer 1

The simplest approach would be an ANOVA, followed by post-hoc t tests.

However, this would ignore that students in the same school will be more similar than students from different schools, even if the students from the same school have different teachers. Andrew Gelman would propose a multilevel Bayesian model... but I would recommend a mixed linear model, also known as repeated measurements ANOVA (we are repeating measurements from the different schools and the different teachers), which is easier to understand and to communicate to non-Bayesians. Look into the documentation of your favorite statistics package. Good luck!

Answer 2

Regarding the last part of your question, the technique you are looking for is called analysis of variance or ANOVA but it is not going to help you here.

You can look at ANOVA as an “adaptation” of Student's t-test for multiple groups. ANOVA is a complex topic and covers many different models but if you run a regular one-way between-subject ANOVA in a two-group setting in which you could also use a t-test, you will in fact notice that the F statistic from the ANOVA is equal to the square of the t-statistic and the respective p values are equal.

The reason why it is not applicable to your example is that your observations cannot be assumed to be independent (e.g. some but not all pupils at a school will share the same teacher) and this independence is one of the major assumptions of the F-Test used in ANOVA. There is also most likely some dependency between schools and teachers (e.g. if each teacher is teaching in one school and one school only, teachers are said to be nested within schools). Educational researchers use something called “multilevel models” to deal with these issues but this is really a lot to handle if this is all new to you.

At least two other issues also need to be considered before jumping into the analysis:

• 10000 is a lot of observations. It's perfectly possible, likely even, that you will find some “significant” differences that really don't mean all that much. You have to ask yourself what counts as a meaningful difference in performance and look at the notions of “power” and “effect size” to better understand this issue.
• If you want a meaningful significance test, your analysis should probably include all the schools in your sample. Looking at the means and then only formally comparing a handful of schools that look different from each other is a very questionable approach.

The problem you have in mind is really more complex than it looks, it could be very useful for you to look for a statistician or at least an experienced researcher in your field that might be able to help you with your analysis.