Understanding regression to mean

As far as I can understand regression to mean is that if I make a measurement, say the mean of the test scores of some students, when the measurement is repeated with the same student with same conditions, the mean of the test scores will be shifted toward the mean. Ideally, the mean in here is the case where the #of student goes to $$\infty$$.

Since each measurement can be seen as independent random variable, by the central limit theorem, the average of those measurements should be the mentioned mean in the limit, so that is why different measurement in a sense "shift" toward the mean in each time.

Is my understand correct ? If not, how can one understand this phenomena in an intuitive way ?

I'll present an example to try to explain the regression to the mean.

Let's assume that we have a random variable $$X \sim N(0;1)$$ that corresponds to the first measurement. And let $$Y \sim N(0;1)$$ be the random variable corresponding to the second measurement, independent to the first measurement.

So, each student will be characterized by a pair of values $$(x_j, y_j)$$, with the expectations of the measurements $$\mathbb{E}(x_j) = 0$$ and $$\mathbb{E}(y_j) = 0$$. If we select the students that have a positive grade in their first measurement, i.e, those with $$x_j > 0$$ (blue circles on the figure), we may be tempted to think that these are the best students. However, if we compute the average of their second grade $$y_j$$ we should find that it will be close to zero, which is the expected value of the entire set of students. Likewise, the average second grade of the other students (black circles on the figure), that got $$x_j \leq 0$$ should also be zero.

Of course that this is a very extreme case, where the second measurement is independent of the first. What if the second measurement was related with the first measurement in a linear way, e.g, let's assume that the second measurement is $$w_j = \alpha x_j + (1-\alpha) y_j$$ for some $$\alpha \in [0; 1)$$. The case $$\alpha = 0$$ is the previous case, and if $$\alpha$$ is close to $$1$$, $$w_j$$ will be close to $$x_j$$.

If we select the best performing students, i.e., those with a positive initial grade, $$x_j$$ (again in blue), the expectation of their final grade should be: $$\mathbb{E}(w_j | x_j > 0) \\ = \mathbb{E}(\alpha x_j + (1-\alpha) y_j | x_j > 0) \\ = \alpha \mathbb{E}(x_j | x_j > 0) + (1 - \alpha) \mathbb{E}(y_j | x_j > 0) \\ = \alpha \mathbb{E}(x_j | x_j > 0) + (1 - \alpha) 0 \\ < \mathbb{E}(x_j | x_j > 0)$$

So, in this example the expectation of the second measurement is closer to zero than the expectation of the first measurement. So, the expectation of the second measurement is closer to the expectation for the entire set of students, that was zero.

– Our
Apr 25, 2019 at 4:59
• You're welcome. Note that I've just presented some examples, not a proof. Apr 25, 2019 at 13:35
• Yes, I know, but that wasn't what I was looking for, so no problem.
– Our
Apr 25, 2019 at 17:18

Your understanding of the effect isn't quite right. If these test scores are random variables that are conditionally independent over time (given their student-wise means), then across all students we expect as many scores to increase or decrease on the second measurement (i.e. about 50% do better the second time and 50% do worse). But consider what happens if we look at the top 1% of students from the first test. Some of their good performance is likely due to stable factors like their intelligence and proficiency at the subject, and this will carry over from one test to the next. That is, these factors determine their mean performance. But part of their good scores on the first test is going to be due to luck. Statistically, this means that the random variability in their test scores turned out to their advantage on this particular test. This puts them in the upper tail of the distribution for that test. But this luck does not carry over to the next test. They could get lucky again, and this probability hasn't changed, but overall it's more likely that they will have average luck (because that's the most likely outcome for everyone). So on the next test, the scores of these students are likely to be close to their average performance (as they typically are). Only because we singled them out based on their previous performance, it seems like these great scorers have a tendency to do worse on their second test.

The opposite effect happens if we singled out the worst 1% from the first test. They likely got pretty bad luck on that first test, but given another try most of them will have average luck (because, again, average luck is more likely overall), so it will seem like their scores have a tendency to increase.

If we look at the test scores from all students though, we will see that they have the same distribution for the first and second tests, with the same proportion of values near the mean or far away from it.

In short, random variables with typical distributions (e.g. Normal) are always more likely to take on values close to their means, whether you look at their first, second or N-th observation. Occasionally, though, they'll take on a value far from the mean, somewhere in the tails of their distribution. If you look only at those times, then it seems like whenever they are very high, they tend to be lower the next time, and vice versa. But this effect doesn't really exist, and the real explanation is that values near the mean are always more likely.