# Regression to the mean and its formula in R

In this book, the estimate of the regression to the mean phenomenon is said to be: $$Prm=100(1-r)$$.

Where $$Prm$$ is the percent of regression to the mean, and $$r$$ is the correlation between the two measures (e.g., pre-, and post-test)

In other words, $$Prm$$ is the distance from the no-regression point to the mean of the population to which the sample mean in question shifts.

Question: Where does this formula come from? It is possible to demonstrate it in an R simulation?

Could the following (sampling distribution of mean differences) be used for such an R demonstration?

## See Update below the code.

library(MASS)

reg.to.mean <- function(n = 50, r = .5, effect.size = 1){

r.matix <- matrix(c(1, r, r, 1), 2)

xy <- MASS::mvrnorm(n = n, Sigma = r.matix, mu = c(0, effect.size))
x <- xy[,1]
y <- xy[,2]

return(mean(y)-mean(x))
}

# Example of use:
samp.dis.mdif <- replicate(1e4, reg.to.mean(n = 100, r = .5, effect.size = 1))

plot(density(samp.dis.mdif))


UPDATE BASED ON THE ANSWER BY @sergio (below):

Dear Sergio, many thanks!

In plain language, I understand that $$(1-r) = (1-\beta)$$. Therefore, we know the relationship between $$x$$ and $$y$$ can be written as: $$y=(1−r)\mu+xr$$. From this equation, I can understand: If $$r = 1$$, then $$x=y$$. If $$r=0$$, then, $$y=\mu$$. For any other value of $$r$$, the relationship between $$x$$ and $$y$$ on a scatterplot will have an intercept amounting to $$(1-r)\mu$$ and a slope amounting to $$r$$.

So, would you please describe (possibly in words) how the last part of my description above implies that mean of $$y$$ (post-test) will shift to the mean of $$x$$ (pre-test, but here their common mean as $$E[y] = E[x]$$) by $$(1-r)$$?

• Please avoid using comments to try to ping people to answer other questions. Jul 26, 2020 at 22:54

Part 1

If $$y=\alpha+x\beta+\epsilon$$ and $$E[\epsilon]=0$$, then $$E[y]=\alpha+E[x]\beta$$, therefore $$\alpha = E[y]-E[x]\beta$$.
If you assume that $$y$$ and $$x$$ have the same marginal distribution, then $$E[y]=E[x]=\mu$$, and $$\alpha=(1-\beta)\mu,\qquad y=(1-\beta)\mu+x\beta$$ i.e. $$y$$ is a weighted average of the population mean $$\mu$$ and the predictor.
But if $$\text{var}(y)=\text{var}(x)$$, then $$\beta=\frac{\text{cov}(x,y)}{\text{var}(x)}=\text{corr}(x,y)$$. This is where $$(1-r)$$ comes from.
Since $$\beta<1$$ (unless $$\beta$$=1, a degenerate case), $$y$$ is on average closer to the population average than $$x$$.

Galton observed that extreme heights in parents are not passed on completely to their offspring. Rather, the heights in the offspring regress towards the population average. Galton called this phenomenon regression towards mediocrity.

Let's use Galton's data (https://github.com/vincentarelbundock/Rdatasets/raw/master/csv/mosaicData/Galton.csv):

> h <- read.csv("Galton.csv")
X family father mother sex height nkids
1 1      1   78.5   67.0   M   73.2     4
2 2      1   78.5   67.0   F   69.2     4
3 3      1   78.5   67.0   F   69.0     4
4 4      1   78.5   67.0   F   69.0     4
5 5      2   75.5   66.5   M   73.5     4
6 6      2   75.5   66.5   M   72.5     4


Since men are taller than women on average, Galton replaced the two parents' heights with the average of the father's and mother's heights, the latter scaled up by $$1.08$$, and multiplyed daughters' heights by $$1.08$$ (see here):

> h$$midparent <- (h$$father+h$$mother*1.08)/2 > h$$midchild <- ifelse(h$$sex=='M', h$$height, h$$height*1.08) > mean(h$$midparent)
[1] 69.22201
> mean(h$midchild) [1] 69.23371  Notice that you can assume $$E[y]=E[x]=69.2$$. You can now use several models, but let's keep it simple: > reg <- lm(midchild ~ midparent, data=h) # simple linear regression > reg$$coefficients (Intercept) midparent 18.7669821 0.7290562 > alpha <- as.numeric(reg$$coefficients[1]) > beta <- as.numeric(reg$coefficients[2])


If the average of the father's and mother's heights is above population average, then their children will be a bit shorter:

> x <- 75              # midparent height
> alpha + beta * x     # average children height
[1] 73.4462


What if $$x$$ is lesser than $$\overline{x}$$? Their children will be a bit taller:

> x <- 65
> alpha + beta * x
[1] 66.15563


This is the regression to the mean.

Part 2

Let's come to pre-test and post-test. Follow David R. Shanks and generate the following data:

> s <- rnorm(100, 1, 1)            # stimulus
> ex <- rnorm(100)                 # random error
> ey <- rnorm(100)                 # random error
> x <- 100 * s + 30 * ex           # performance score
> y <- 0.30 * s + ey               # awareness measure
> (r <- cor(x,y))                  # y and x are less than perfectly correlated
[1] 0.352693
> summary(lm(y ~ x))$coefficients # y = a + bx + e Estimate Std. Error t value Pr(>|t|) (Intercept) 0.137141042 0.1436701078 0.9545552 0.3421513684 x 0.003686547 0.0009880176 3.7312569 0.0003194889  $$s$$ is a stimulus, $$x=100s+30\epsilon_x$$ is a performance score, some assessment of perception or memory, $$y=0.30s+\epsilon_y$$ is an awareness measure, such as a verbal report or a forced-choice response (read Shanks's paper to better understand the experiment, I'm not a psychologist :) Now convert $$y$$ and $$x$$ to standard normal variables $$z(y)$$ and $$z(x)$$, which have the same (marginal) distribution: > zx <- scale(x) # mean = 0, sd = 1 > zy <- scale(y) # mean = 0, sd = 1 > (r <- as.vector(cor(x,y))) # unchanged [1] 0.352693 > summary(lm(zy ~ zx))$coefficients
Estimate Std. Error      t value     Pr(>|t|)
(Intercept) 1.729815e-17 0.09405011 1.839248e-16 1.0000000000
zx          3.526930e-01 0.09452392 3.731257e+00 0.0003194889


Notice that $$\hat\beta=r$$.

When performance and awareness measures are collected, usually performance show some above-chance sensitity to $$s$$, but awareness does not. This is why researchers have turned to so-called post hoc selection: data from partecipants whose awareness measure is below some cutoff are analyzed separately. If their mean performance score is greater than an appropriate baseline for that test, then it is concluded that some true unconscious cognition has been demonstrated.

Actually, post hoc analysy does not show anything else than regression to the mean. Let $$c$$ be a cutoff on $$zy$$, e.g. $$c=\overline{zy}=0$$. If you select $$zy$$ values lesser than $$c$$, and the corresponding $$zx$$ values, you'll always get $$zx$$ values closer to the mean than $$zy$$ values:

> cutoff <- mean(zy)                # cutoff = 0, you can change it
> y_rtm <- zy[which(zy <= cutoff)]  # awareness measure <= cutoff
> x_rtm <- zx[which(zy <= cutoff)]  # corresponding performance score
> mean(y_rtm)                       # low awareness measure
[1] -0.7361932
> mean(x_rtm)                       # performance score closer to the mean
[1] -0.266041


The same result if you select $$zy$$ values greater than $$c$$:

> y_rtm <- zy[which(zy >= cutoff)]  # awareness measure <= cutoff
> x_rtm <- zx[which(zy >= cutoff)]  # corresponding performance score
> mean(y_rtm)                       # high awareness measure
[1] 0.7361932
> mean(x_rtm)                       # performance score closer to the mean
[1] 0.266041


Another example of regression to the mean. It's just like selecting shorter parents (parents whose height is below some cutoff) and observing taller children, or selecting taller parents and observing shorter children.

Clearly, the same happens if you use nonscaled variables: "When two variables are imperfectly correlated [...] it is a statistical certainty that applying an extreme cutoff on one dimension [...] will yield a less extreme cutoff for the expected value of the other variable" (Shanks). Scaled variable help understand that the extent of regression to the mean depends on the correlation $$r$$, because when scaling (generally speaking, when $$y$$ and $$x$$ have the same distribution) $$\hat\beta=r$$.

So, would you please describe (possibly in words) how the last part of my description above implies that mean of $$y$$ (post-test) will shift to the mean of $$x$$ (pre-test, but here their common mean as $$E[y]=E[x]$$) by $$(1−r)$$?

Look at Galton's data. You have $$E[y]=E[x]=69.2$$, and:
a) if $$x=75$$, then $$\hat y=\hat\alpha+\hat\beta 75\approx 73.4\approx 75+(1-r)(69.2-75)$$;
b) if $$x=65$$, then $$\hat y=\hat\alpha+\hat\beta 65\approx 66.1\approx 65+(1-r)(69.2-65)$$.

where $$r=\hat\beta$$. The distance between $$x$$ and $$\mu$$ is $$|\mu-x|$$, $$\hat y$$ gets closer to $$\mu$$ by $$(1-r)|\mu-x|$$. Why? Because: \begin{align*} \hat\alpha+x\hat\beta &= (1-\hat\beta)\mu+x\hat\beta \\ &=\mu-\mu\hat\beta+x\hat\beta+x-x =x+(1-r)\mu-(1-r)x\\ &=x+(1-r)(\mu-x) \end{align*}

References

Bruce E. Hansen, Econometrics, §2.26
Stephen M. Stigler, The History of Statistics: The Measurement of Uncertainty before 1900, Chap. 8
James A. Hanley, "Transmuting" Women into Men: Galton’s Family Data on Human Stature, The American Statistician, 2004, 58(3). Hanley compares sharper models. David R. Shanks, Regressive research: The pitfalls of post hoc data selectionin the study of unconscious mental processes, Psychonomic Bulletin & Review, 2017, 24.

• (a) $r$ is "the correlation between the two measures", $\beta=r=\text{corr}(x,y)$; (b) if $x_i>\overline{x}$ then $y_i<x_i$, if $x_i<\overline{x}$ then $y_i>x_i$, so $|y_i-\overline{x}|<|x_i-\overline{x}|$, $y_i$ is closer to $\overline{x}$; (c) in an experimental study you have pre-test and post-test, in an observational study you have parents and children, a sequence of profits in 10 years (see Hansen) etc. (d) yes, $x$ and $y$ belong the the same population, a stable population, so their means and their variances are equal. Jul 27, 2020 at 7:07
• "Regression to the mean is a necessary feature of stable distributions" (Hansen, §2.26, last sentence). Jul 27, 2020 at 7:12
• @rnorouzian I'added a second part to my answer. HTH. Jul 27, 2020 at 11:22
• @rnorouzian I've added a sub-answer :) Jul 27, 2020 at 22:47
• I must agree with the OP that this answer needs refinement to show very clearly where regression to the mean occurs. I'm super confused by your answer too. What is the result of your Galton regression analysis regarding the regression to the mean i.e., exactly what has regressed to what? please annotate your code. Why all of a sudden, you show What if x is greater than x_bar = 69.2? what does that show? What are the role of scaling up mother's height and daughters' height but not sons and fathers?
– Reza
Jul 27, 2020 at 23:51