What's complicated about regression to the mean?

Question

Note: I am a bit of a novice when it comes to statistics and data analysis.

Reading the chapter on regression to the mean in Kahneman's Thinking Fast and Slow, I came across the following passage:

The very idea of regression to the mean is alien and difficult to communicate and comprehend. Galton had a hard time before he understood it. Many statistics teachers dread the class in which the topic comes up, and their students often end up with only a vague understanding of this crucial concept.

I really don't see what's difficult/complicated to explain or counter-intuitive, only that the concept might "hide" and be easy to miss in certain situations. But, the statement "Many statistics teachers dread the class..." make me actually believe that I have misunderstood something.

Made simple, here's my intuitive understanding: If we observed an outcome that had a very low probability of happening, then the next time we do the same thing again, the outcome will not be as extreme. To me this is self-evident, since by definition, the first event was unlikely. If a bunch of people throw a "gaussian die" with values 1-10, and some throw 1 or 10, you don't expect them to repeat it the next time for the same reason you didn't expect them to do it the first time. This will then of course be more or less prominent depending on how much randomness (luck) is involved.

Question: Is my intuitive understanding correct or am I missing something?

I don't want to sound like I think I'm smarter than e.g. any statistics teachers, quite the contrary; I genuinely believe I've missed something here.

Answer 1

I think your intuitive understanding is correct. As for why it poses a conceptual problem for so many, I can only offer a guess.

Most statistical procedures are designed around means: comparing means of groups, evaluating how the mean of one variable changes across gradients in other variables, etc.

But we are occasionally very interested in evaluating hypotheses about or predicting extremes: the fastest runner, the best-performing stock, etc.

Adapting the statistical procedures and mental toolkit/shortcuts we have developed that is based on means to deal with extremes can be nontrivial. It's often not as simple as doing a quantile regression, because the baseline itself has poor (or perhaps 'unstable' is a better term) statistical properties, as you note in your question. If people have not grappled with the importance of this baseline problem before, it can be challenging to understand because it is fairly fundamental and requires revisiting (and revising) some basic ideas. This can be especially challenging when dealing with time series, which poses its own distinct set of challenges even when we are not interested in extremes.

In my experience, even experienced scientists can struggle with the idea. The fact that this seems obvious to you is excellent news - you have developed a mental toolkit which makes it easier for you to deal with this. Often, exposure to concepts like these early on in one's training can make otherwise difficult concepts clear. Choosing when to introduce ideas like this is an important challenge when designing courses for students: too early and it can confuse and seem too abstract, too late and it risks taking extra effort because of having to undo earlier mental models.

EDIT:

To add a realistic example that might shed some light on this.

Imagine a business that tests all its employees for performance on some skill. The results show that there is clear room for improvement and the business could make some money by improving those skills. It would be too disruptive to retrain everyone, so instead it takes the people who score in the lowest 10% and puts them through extra training.

At the end of the training, this group is tested again. And voila - their average score is higher than it was in the original test! The training has worked, the business has made a useful investment, and should now make more money - right? Not necessarily. If you understand regression to the mean, you would realise that just testing the bottom 10% a second time should lead to a higher average score, even without any additional training. This is because these individuals' position in the bottom 10% on the first test is because of a combination of (probably poor) ability and random factors that happen to reduce performance in that test. Since they were in the bottom 10%, those random factors were likely more negative in this instance; test them again and the random factors would probably not be quite so negative. They would seem to improve in performance, but it would be illusory.

The crucial step here was the selection of the extreme values for extra training/testing - but this decision is entirely understandable! It's just that it's easy to forget that inference and prediction are much trickier when you have selected an extreme sample (or a non-random sample more generally). Put yourself in the place of a businessperson trying to improve performance, or even a student trained in simple statistical procedures - wouldn't every action taken by the business in the example seem reasonable?

Answer 2

Your explanation is accurate, though I think there is an essential detail it doesn't address. Namely that in the typical confusing situation, the recent outcome does give you information about the new outcome. And regression to the mean is still present.

Think about parent's height vs children's height or a sports team's performance in the first segment of a game vs the second segment. There is some underlying factor (genes, team's current fitness) that is relatively constant and influences both outcomes. But there is also some random factor (environment / gene expression, luck in sports) that influences each outcome separately. As long as there is any random factor, regression to the mean is present. Yet people intuitively overlook that if they are thinking about the constant factor.

This may be a consequence of people's general tendency to see patterns where none exist and therefore underestimate the random component.

In the example with the sports team that performed horrible in the first segment, the trainer will yell at the team in the break, the team will perform better despite the intimidation, and the trainer will think that the yelling was helpful. It's a false pattern the trainer sees and the trainer's belief in it is reinforced by the regression of the performance toward the mean. (In that situation the team's fitness/spirit is to some degree not necessarily a constant / the same in each segment, but it's still not the most important explanation for performance differences.)

In case you are interested, here's a toy example. An approximate version of the "Gaussian" die you mention does exist, it's called "throwing two dice and computing the sum". Imagine I would do that and tell you only the sum. Then I would take the die to my right and throw it again and leave the left die untouched. What do you expect the new sum to be? If the old sum was, say, an 11, you'd knew that my dice had shown 5 and 6. So it is reasonable for you to expect a rather high number for the new sum, since one of the dice is left untouched. That is what people often intuitively get right. At the same time it is reasonable for you to expect the new sum to be lower that 11, since one of the dice is thrown again. That is the regression toward the mean that people often miss in more complicated situations.

Tangential topic: Bayesian inference

As pointed out in Sextus Empiricus' answer, there is a relation to Bayesian inference. The uncertainty about the right die's result (random component) may be called aleatoric, while the uncertainty about the left die (fixed component) may be called epistemic. Since the left die is already determined before the second roll and we can gain knowledge about it from the reported sum in the first roll. This process of gaining/inferring probabilistic knowledge is called Bayesian inference and works roughly as follows in the above example.

• We know that the number was sampled with uniform probability between 1 and 6 ("prior").
• We observe a value of 11 and notice that any number other than 5 and 6 would make it impossible (chance of zero) to observe 11 while both 5 and 6 would produce a chance of 1/6 ("evidence").
• We conclude (but usually have to compute) that the left die must show a 5 or a 6 and both have equal probability ("posterior"), since they had equal prior and equal likelihood.

From that posterior over the left die and the known distribution over the right die, we can calculate the distribution over the new sum.

The following image from this website shows how the quantities in Bayesian inference would look like if we didn't have fair dice but Gaussian "dice". The fact that the posterior is not as far out as the evidence corresponds to regression toward the mean. The fact that it is away from the prior shows that the regression does not go all the way to the (prior) mean. (A child of tall parents will usually still be higher than average.) The amount of the regression depends on the relative influence of the random component (width of evidence curve) and the fixed component (width of prior distribution). This answer on Cross Validated has an image that shows how these widths influence the posterior.

Answer 3

"What's complicated about regression to the mean" in most realistic cases is that, unlike rolling dice of completely known characteristics, (a) previous data points usually give us information about future data points, e.g. by allowing us to make (or improve upon) estimates of the statistical parameters of the underlying data-generating process (DGP), and (b) the information we gain about the DGP generally leads us to expect "like will be followed by like". For example, the taller I know a father is, then based on the genetic heritability of height, the taller I should expect his son to be. This is obviously in pedagogical tension with most people's intuition of regression to the mean: if heights of fathers and sons were independent random variables, as in your dice experiment, then a tall father is expected to be followed by a shorter* son, and indeed for each inch taller the father is, then the expected height difference between them grows by one inch also.

The resolution to this tension requires me to synthesise these two apparently contradictory effects into a more sophisticated mental model: one in which knowing a father is tall leads me to expect his son will be tall too (because of what I've gleaned about genetic factors influencing the son's height), but probably not as tall as the father (due to regression to the mean). For example, a son-versus-father regression line with slope between zero and one would be consistent with both these facts. For simplicity let us assume mean height $m$ is consistent across generations, so the son's expected height $\mathbb{E}(y)$ and father's height $x$ are related by:

$$ \mathbb{E}(y) - m = \beta (x-m),\quad 0 < \beta < 1 $$

Then for every inch taller that the father is, the son's expected height $\mathbb{E}(y)$ increases by $\beta$ inches, a less than one-for-one rise. A tall father is expected to be followed by a son who is tall both in absolute terms and relative to the average height. Yet the extent to which the son is expected to be above average height by, $\mathbb{E}(y) - m$, is less (since we multiply by a number between zero and one) than his father stood above the average by, $x - m$. Relative to the father, the expected height differential is:

$$ \mathbb{E}(y) - x = -\gamma x + \gamma m,\quad \gamma = 1 - \beta, \quad 0 < \gamma < 1 $$

That is, knowing that a father is one inch taller, leads me to expect the son to be an additional $\gamma = 1 - \beta$ inches shorter compared to his father. Note that in the above formula the expected father-son height differential is zero when $x=m$: there is no "regression towards the mean" effect if we're at the mean already. Also, if we put $\beta = 0$ so $\gamma = 1$, you can see we recover the case discussed above where the son's height is unrelated to his father's. The son's expected height is simply the mean height, and each additional inch of height for a taller-than-average father leads us to expect the son will be one more inch shorter than him.

Perhaps the above seems trivial to you, but I hope it illustrates why the mental gymnastics required are more strenuous than the dice example suggests. And even this is really just a jumping-off point: why, if this procedure is iterated over many generations, do all the great-great-...-grandchildren not converge towards the average height? To understand this, we need to extend our model to show the whole distribution of heights at each generation. What if we allow for there to be an underlying trend in mean height over generations, with mean height rising due to improved population health and nutrition?

If you use independent rolls of a 0-10 di(c)e as your mental model for regression to the mean, you miss out on a lot of complexity. (You called the die "Gaussian" which contradicts its distribution of scores being over a bounded interval, but I'll assume you just mean "central values are more likely, with probability falling off in the tails", like a symmetric Beta distribution with $\alpha = \beta > 1$ that's been rescaled and translated as necessary.) If someone scores a 9 they're well into the upper tail and very likely to do worse the next time. But what if there are two possible dice and you don't know which one they are using? Let's say the dice run 0-10 ("low" with mean 5) and 5-15 ("high" with mean 10). If initially we rate it as a 50:50 chance whether the the low or high die is being used, the expected score is 7.5 so a 9 is still a higher-than-average score. But our calculus for what we expect to happen on the next roll must take into account that a score of 9 is indicative evidence — albeit not definitive — that the higher-scoring of the two dice is in play. Indeed a score of 11 would have told us this for sure, and the expected score on the next roll would clearly be 10. For 9, we could update our assessment of the probability we assigned to each die using a Bayesian method, as suggested in some of the other answers here, and work out the next roll's expected score from there.

If we replace the high/low dichotomy by a whole spectrum of possible distributions, this resembles many real-life situations where previous data hints what distribution is being sampled from. Our response to a student acing a test, or a runner's great time in a race, can't simply be "well, they're unlikely to repeat that kind of success". We must also incorporate the evidence we've seen that we have encountered a student or runner of high ability, and update our expectations of their future performance accordingly. If our student is a fighter pilot in training, as Kahneman discusses, the effect of trends is important, since (hopefully!) we would not expect their performance to remain the same over the course of their training. If we learn a pilot has previously had a run of high scores, does that lead us to expect the next score will be (i) higher since we have evidence suggesting the pilot has high ability, (ii) lower due to regression to the mean, (iii) higher since with additional training the pilot's scores should trend upwards, or (iv) some combination of the above effects? (Worth pondering the implicit higher/lower than what?) And this is before we consider that some pilots will be faster learners than others, that individual trends could be estimated from each pilot's previous testing data, and that a pilot with an apparently outstanding record of improvement to date might be a brilliantly fast learner yet might also be about to hit the regression to the mean effect on their trend, not just on their level...

Two final points re "what's complicated". Plenty of people think they understand regression to the mean because "obviously if a student's aced ten tests in a row, or a runner's had a series of good times, they must be overdue a bad result". In other words, they have conflated regression to the mean with the gambler's fallacy. I think this is the most common misconception encountered when teaching the topic. However, it isn't the most dangerous. As Kahneman points out, there's a particular problem when you anticipate a trend or change in level due to interventions targeted on the basis of previous performance. Suppose you scold pilots who performed badly on their last training mission, and praise those who did well. Even if these interventions have no effect at all, we would expect the scolded group to improve on their previous performance and the praised group to get worse, purely due to regression to the mean. Similarly, set up an intervention group for students who did poorly in a test at school: give them extra tuition, or feed them more fresh fruit or fish oil, or just teach them to stand on one leg for thirty seconds before exams. Their performance in the next test (both relative to their previous performance and relative to the rest of the class) is likely to improve. I scarcely need to mention the potential for misleading effects in medical studies which pick out high-risk groups as a target for (what's intended to be a) protective intervention.

The problem is not only that we fool ourselves when we observe how our brilliant intervention for low-performers "succeeded" (or for high-performers "failed counterproductively"). We can "fool" the standard statistical tests too: SPSS/Stata/R will quite likely churn out a highly significant p-value when you compare the intervention group's before/after scores, or compare the intervention group's improvement to the non-intervention group's improvement. The issue is we are making the wrong comparison: ideally we need a control group who were eligible for the intervention but were randomly assigned not to receive it. The control group should experience similar regression to the mean effects as the intervention group, but not the (dis)advantages of the intervention itself, allowing these effects to be distinguished. But unless we are undertaking a proper study with some intent to write it up, how often would we do this? Would it even be ethical if the intervention was believed to be beneficial? A class teacher tasked with ensuring all students made the required grade would be most unlikely to refuse extra support to a randomly chosen subset of low-achieving students just to confirm the effectiveness of their methods: as a result, it's quite likely the teacher's impression of the value of their intervention is over-inflated. Even people who understand the maths of regression to the mean are prone to this kind of error, a trap we likely fall into daily...

---

($*$) "Shorter" compared to the father, not in absolute terms or relative to the average height! Thinking relative to the correct reference level is important here, and a point of confusion in itself — certainly not something all students get right.

Answer 4

Relationship with bayesian inference.

To make it complicated: regression towards the mean can be related to Bayesian inference.

If we estimate an $\alpha \%$ confidence interval for some parameter then this confidence interval will not contain the true parameter with $\alpha \%$ probability when we condition on the observation (see Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?).

On the other hand, an $\alpha \%$ credible interval will contain the parameter with $\alpha \%$ probability (if the prior is correct/reasonable) and it does this by making estimations that are closer to the population mean.

Some critical notes to your dice example

If we observed an outcome that had a very low probability of happening, then the next time we do the same thing again, the outcome will not be as extreme. To me this is self-evident, since by definition, the first event was unlikely. If a bunch of people throw a "gaussian die" with values 1-10, and some throw 1 or 10, you don't expect them to repeat it the next time for the same reason you didn't expect them to do it the first time.

In your example you are not speaking about parameter estimates or predictions based on parameter estimates.

You have a random dice roll. It is obvious that it will be different every time.

Instead, you should have this Gaussian dice roll be depending on some parameter, let the rolls from a dice be distributed as $\mathcal{N}(\mu,1)$ and this parameter $\mu$ is unknown. If somebody rolls 10, what will be your guess for the next roll by the person? Probably it should be close to $10$ again, but if you know the distribution of $\mu$ for the population then you can make a much better estimate by using a Bayesian estimate (which will be something that is closer to the population mean of $\mu$).

The dice roll example that you used, that would make students understand that if we got an observation above $\mu$ then the next time the observations will be likely lower, and if we got an observation below $\mu$ then the next time the observations will be likely higher.

But what sort observation did we have. Did we observe above or below $\mu$?

With the dice example it makes sense that the observations shift towards the parameter $\mu$ associated with the individual. But why should it be such that our estimates regress toward the population mean? If the parameter $\mu$ happens to be above the population mean, then why should we get more often an observation like $\mu+10$ and overestimate instead of an observation like $\mu-10$ and have a mean that is underestimated?

The dice example explains why extreme observation will move towards the mean of an individual $\mu$. We can have observations above $\mu$ and observations below $\mu$ like with your dice.

But the dice example does not explain why these extreme observations are more often away from the population mean, such that we observe regression towards the population mean.

Regression away from the mean

To show how it can be more subtle, it is possible to create an example where there is regression away from the mean conditional on the measurement.

Let $\mu_i$ be a mixture distribution with fifty fifty two Gaussian distributions, one with mean -5 and the other with mean 5 and deviation 1.

Let a first observation $x_i$ and a second observation $y_i$ be distributed as Gaussian with mean $\mu$ and deviation 1.

Then for a first observation between -2 and 2, there will be mostly regression away from the mean.

Answer 5

I think part of the problem is the phrase "regression to the mean". It encourages people to believe the gambler's fallacy. Suppose you flip a coin 10 times and get heads every time. It is an unlikely event, 1/1024 but quite possible. People think regression to the mean means that heads are less likely going forward. In fact it means this excess of heads gets swamped as we increase the number of trials. As long as we still believe it is a fair coin, we expect 5 heads in the next 10 trials. We expect 500 in the next 1000. If we flip another 10,000 times, the most likely total number of heads is 5,010. The standard deviation is 50, so the excess of 10 is nothing.

The hard part is after the first 10 flips do you still believe the coin is fair?