30
$\begingroup$

My professor wrote:

Research using sophisticated statistical techniques indicates that teaching expertise accounts for about 30 percent of the variance in student achievement (Hattie, 2003). Think about what your student test scores would look like if you could achieve this 30 percent increase with each group of students that passed through your classes each year.

I believe the professor misunderstands variance accounted for. "Teaching expertise accounts for about 30 percent of the variance in student achievement" means that a teacher is responsible for 30% of what a student achieves. You can't infer from this that a great teacher will achieve a 30% increase in student achievement relative to... a teacher who didn't teach at all? If we had 2 teachers, one at the bottom of teaching expertise and the other at the top, teach classes which had all other variables constant (socioeconomics, motivation, etc.), would the better teacher's class achieve a 30% increase relative to the worse teacher?

$\endgroup$
  • 1
    $\begingroup$ The current answers are great in giving insight why the %variance explained does not coincide with %increase achievement (it depends on the amount of difference between the teachers, and as Stephan explains the %increase is also ambiguous/relative due to the intercept). I want to add that we must also think about the issue causality vs correlation. The quote is not so explicit about it, but we should not interpret the 30% like the teaching expertise is causing that part of the student's achievements. It is not like a change to more expertise will change a student's achievement by 30%. $\endgroup$ – Sextus Empiricus Jul 28 at 20:51
  • 1
    $\begingroup$ "if you could achieve this 30 percent increase" This is an ambiguous statement. 30 percent, relative to what? Currently you seem to interpret it as 30% difference between students from teacher 1 versus teachers from teacher 2 (like the one group of students will do 30% better relative to the other group). But that is not the necessary interpretation of 30 percent increase. In the case of the correlation here the 30 percent increase means that for each unit of teacher expertise the students have 0.3 units increase of achievement. That is also a way to see 30% increase. $\endgroup$ – Sextus Empiricus Jul 28 at 20:56
  • $\begingroup$ Do you have the Hattie, 2003 reference with details like title? Their definition of "expertise" may mean "tenure" or the likes, because expertise does not seem to be quantifiable. $\endgroup$ – user3819867 Jul 29 at 10:00
  • $\begingroup$ I note that the quoted portion does not claim that better teachers produce better student outcomes. Better teachers reducing student outcome is compatible with the quote. That is, "accounts for about 30 percent of the variance" doesn't tell you the signs of anything. $\endgroup$ – Eric Towers Jul 30 at 15:34
  • $\begingroup$ I'm always cynical about this type of statements. It's like saying one additional hour of sleep adds 5 years of life. The problem is that there's a ton of these "X amount of Z adds Y number of years to life," so if I do all of these Zs, will I live 500 years? Obviously, not happening. $\endgroup$ – Aksakal Jul 30 at 19:01
39
$\begingroup$

You are right in suspecting that your professor misunderstood.

The correct answer is that we cannot say anything whatsoever about the percentage improvement in student achievement driven by teacher expertise. Nothing at all.

Why is this so? The quote is in terms of variance explained. Variance explained has nothing to do with the actual values on which the scales are measured - which any percentage improvement in student achievement would be accounted in. The two are completely separate.

Let's look at an example. Here is some simulated data:

variance_explained

R code:

nn <- 1e2
set.seed(1) # for reproducibility

teaching_expertise <- runif(nn)
student_achievement <- 5+0.1*teaching_expertise+rnorm(nn,0,0.05)
model <- lm(student_achievement~teaching_expertise)

plot(teaching_expertise,student_achievement,pch=19,las=1,
    xlab="Teaching Expertise",ylab="Student Achievement")
abline(model,col="red")

Note that the model is correctly specified: student achievement depends linearly on teaching expertise, and that is what I am modeling. No cheap tricks here.

We have $R^2=0.30$, so teaching expertise indeed accounts for 30% of student achievement (see here):

> summary(model)

Call:
lm(formula = student_achievement ~ teaching_expertise)
... snip ...
Multiple R-squared:  0.304,     Adjusted R-squared:  0.2969

However, here is the student achievement we would predict for teachers at the very bottom (teaching expertise of 0) vs. at the very top of the range (1):

> (foo <- predict(model,newdata=data.frame(teaching_expertise=c(0,1))))
       1        2 
4.991034 5.106651

The improvement is on the order of $\frac{5.11-4.99}{4.99}\approx 2.4\%$.

> diff(foo)/foo[1]
         2 
0.02316497

(Plus, this is expected achievement. Actual achievement will be different. With regression to the mean typically being stronger at the extremes, the actual difference will be even smaller.)

And you know what? We could change this percentage change to pretty much any number we want. Even a negative percentage improvement! How? Simply by changing that single innocuous number 5 in the data simulation above, i.e., the intercept.

What's going on? Variance explained measures the amount by which the (sum of squared) residuals are reduced by a model, i.e., the difference between the residuals to the regression line and the residuals to the overall average. By changing the intercept (the 5), we can shift everything up and down. Including the overall average. So changing the intercept will leave variance explained completely unchanged. (If you have R, try this. We'll wait.)

However, shifting everything up and down will change the concrete scores. In particular the percentage improvement of a "good" vs. a "bad" teacher. If we shift everything down far enough, we get a negative student achievement for the "bad" teacher. And a positive change for the "good" teacher against a negative baseline will give you a negative percentage improvement. (Again, try this. An intercept of -1 works.)

Yes, of course such negative percentage improvements make no sense here. This is just an illustration of the fact that there is zero relationship between variance explained and percentage improvement in measurements.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I feel like both answers so far kinda miss the mark. You're basically saying "well if everyone scored above 90% on the test then 30% of variance only accounts for a few percent so the professor is wrong". But that's pretty fallacious. Of course the absolute improvement in performance depends on the assessment metric, and I really doubt that their professor thinks the best teachers could have their students get 120% on that exam. If you look in terms of percentiles then 30% definitely has a valid ststistical meaning. $\endgroup$ – Joseph Ireland Jul 28 at 10:23
  • 1
    $\begingroup$ @JosephIreland: I'm afraid I don't follow your argument. I don't say anything about everyone scoring above 90%. My toy example does not have a ceiling. Of course, one could use achievements that are between 0% and 100%, but that would just make everything more complicated - without changing the underlying logic. $\endgroup$ – Stephan Kolassa Jul 28 at 10:28
  • 7
    $\begingroup$ There's also the fact that 30% of variance explained doesn't tell you anything about the gradient and that there could even be a NEGATIVE association for all the '30% of variance explained' statistic tells you by itself. $\endgroup$ – H. Green Jul 28 at 10:38
  • 1
    $\begingroup$ @JosephIreland: "academic achievement is judged relative to your peers" - not everywhere. If achievement is measured by percentiles, then there are still transformations to give you any result you like. It will just not be as simple as just tweaking the intercept. You will need to play around a bit to get something that gives you the same $R^2$. Which makes the exercise more complicated, to no discernible profit. The fundamental point still is that there is no relationship between variance explained and percentage improvement. $\endgroup$ – Stephan Kolassa Jul 28 at 10:58
  • 2
    $\begingroup$ @JosephIreland: grading on a curve is not the only possibility. And indeed, if the curve is calculated within a class (so, with a single teacher), then assuming better teaching expertise improves everyone's performance equally, then (a) improving expertise will do nothing to relative standing, and (b) the entire concept of variance explained in students' achievement by a single teacher's expertise is completely meaningless in the first place. Again: I made some assumptions to illustrate the point. You may prefer others. I maintain that the main point still holds. $\endgroup$ – Stephan Kolassa Jul 28 at 11:53
19
$\begingroup$

The Hattie 2003 paper mentions a simple form of hierarchical linear modelling ignoring interactions. The paper’s description of the 30% isn’t particularly thorough, with broken links in the references making it difficult to see where the number even came from. I assume his approach relied on partial R-squared.

The answer is no, going from a bad teacher to a good teacher can’t be expected to increase performance by 30%. The two 30%'s are measured completely differently.

For example, suppose performance followed this equation:$$\text{performance} = \beta_0 + \beta_1 ~\text{studentEffort} + \beta_2 ~\text{teacherEffort} + \text{noise}$$If the $\beta_2$ is small, the performance graph would be nearly flat as teacherEffort changed. This can happen no matter what the $R^2$ is or how it might divide up into partial $R^2$'s.

In other words, saying that teachingEffort accounts for 30% of a variation doesn't tell you what that variation is over the dataset, i.e. how much performance changes.

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

You write '"Teaching expertise accounts for about 30 percent of the variance in student achievement" means that a teacher is responsible for 30% of what a student achieves.'

A better formulation would be "A teacher is responsible for 30% of the difference in performance between students".

In other words, if the average performance of some group of students with teacher A is 80 points and the average performance of another group with teacher B is 70 points, the performance of the teachers A and B can account for around 3 points (30% of the 10 point variance in performance).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Still wrong, your example has confused "difference in performance between students" with "difference in mean performance between groups of students" $\endgroup$ – Ben Voigt Jul 29 at 18:08
  • $\begingroup$ How do you think they measured this effect? What experimental design do you propose to investigate what you've just claimed without using groupings of students? As you can see from pages 10-15 of the paper this statistic comes from ( research.acer.edu.au/cgi/… ), they did indeed use four separate groups of students in their study. $\endgroup$ – sintax Jul 30 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.