4
$\begingroup$

So in a simple linear regression

Y = B0 + B1X

B0 and B1 are negatively correlated. When slope goes up, intercept must go down, in order for the regression line to still pass through as much of the data as possible.

My question is, does the same intuition apply for a multiple linear regression assuming all the predictors are independent from each other. Would each slope coefficient, Bi, individually be negatively correlated with intercept, B0?

Thanks!

$\endgroup$
2
  • 2
    $\begingroup$ You need to inform your intuition with correct observations! The negative correlation is not universal. For instance, when the mean of the $x_i$ is negative, the correlation typically is positive. The correlation is guaranteed to be zero when the mean of the $x_i$ is zero. $\endgroup$ – whuber Nov 27 '19 at 18:55
  • $\begingroup$ Thanks, I did not think of that. $\endgroup$ – confused Nov 27 '19 at 19:25
3
$\begingroup$

The question asks if estimated slopes must be correlated with the estimate of the intercept. The answer is no, not necessarily. The slope will not be correlated with the intercept if the sample mean of $X$ is $0$. Below is a simple simulation, coded in R:

set.seed(1145)  # this makes the example exactly reproducible
x = runif(n=50, min=-4, max=+4)
x = x - mean(x)
mean(x)  # [1] 3.680216e-17
xr = x + 5
xl = x - 5
z = 0 + 1*x

out.mat = matrix(NA, nrow=100, ncol=6)
colnames(out.mat) = c("centered int", "centered slope", 
                      "right int", "right slope", 
                      "left int", "left slope")
for(i in 1:100){
  e  = rnorm(50, mean=0, sd=1)
  y  = z + e
  mc = lm(y~x)
  mr = lm(y~xr)
  ml = lm(y~xl)
  out.mat[i,1:2] = coef(mc)
  out.mat[i,3:4] = coef(mr)
  out.mat[i,5:6] = coef(ml)
}
cor(out.mat[,1], out.mat[,2])  # [1] -0.03190039
cor(out.mat[,3], out.mat[,4])  # [1] -0.9135444
cor(out.mat[,5], out.mat[,6])  # [1]  0.9090916

windows(width=9, height=3.5)
  layout(matrix(1:3, nrow=1))
  plot(out.mat[,1], out.mat[,2], xlab="int", ylab="slope", main="Centered")
  plot(out.mat[,3], out.mat[,4], xlab="int", ylab="slope", main="Positive")
  plot(out.mat[,5], out.mat[,6], xlab="int", ylab="slope", main="Negative")

enter image description here


(The original answer, below, pertains to the original phrasing of the question, which specifically asked if a negative correlation between slopes and the intercept was necessary.)

It depends on whether your data are to the right of $0$ on $X$ or to the left (i.e., your $X$ data are all negative). That is less common, it would seem, but there is nothing mathematically necessary about it. If your $X$ data were negative, then there would be a positive correlation between $\hat\beta_0$ and $\hat\beta_1$.

set.seed(1145)  # this makes the example exactly reproducible
x = runif(n=50, min=-4, max=+4)
y = 0 + 1*x + rnorm(50, mean=0, sd=1)
coef(summary(lm(y~x)))
#               Estimate Std. Error    t value     Pr(>|t|)
# (Intercept) 0.05130084 0.12468707  0.4114367 6.825837e-01
# x           0.92041320 0.05673818 16.2221146 4.094011e-21
xr = x + 5  # w/ pos slope, moving x to the right will make the intercept go down
xl = x - 5  # w/ pos slope, moving x to the left  will make the intercept go up
coef(summary(lm(y~xr)))
#               Estimate Std. Error   t value     Pr(>|t|)
# (Intercept) -4.5507652 0.32128386 -14.16431 9.123907e-19
# xr           0.9204132 0.05673818  16.22211 4.094011e-21
coef(summary(lm(y~xl)))
#              Estimate Std. Error  t value     Pr(>|t|)
# (Intercept) 4.6533668 0.29804604 15.61291 1.929555e-20
# xl          0.9204132 0.05673818 16.22211 4.094011e-21

windows()
  layout(matrix(c(1,2,3), nrow=3, byrow=TRUE))
  plot(x, y, xlim=c(-9, 9), ylim=c(-4,4))
  abline(.05, .92, col="red")
  abline(v=0, col="gray");  abline(h=0, col="gray")
  plot(xl, y, xlim=c(-9, 9), ylim=c(-4,4))
  abline(4.65, .92, col="red")
  abline(v=0, col="gray");  abline(h=0, col="gray")
  plot(xr, y, xlim=c(-9, 9), ylim=c(-4,4))
  abline(-4.55, .92, col="red")
  abline(v=0, col="gray");  abline(h=0, col="gray")

enter image description here

$\endgroup$
3
  • $\begingroup$ It would help to draw explicit conclusions about the multiple regression case, because that is the focus of the question. $\endgroup$ – whuber Nov 27 '19 at 19:03
  • $\begingroup$ @rnorouzian, you should ask this as a new question. It shouldn't be buried in comments here. I'm going to delete these comments, but I can answer your question in a new thread when it arrives. $\endgroup$ – gung - Reinstate Monica Jun 29 '20 at 16:20
  • $\begingroup$ @gung-ReinstateMonica, Thank you, HERE is my question. $\endgroup$ – rnorouzian Jun 29 '20 at 16:49
1
$\begingroup$

Correlation of the estimated coefficients in linear regression: For linear regression using OLS estimation of the coefficients, the general formula for the variance of the estimated coefficient vector (implicitly conditional on the explanatory variables) is:

$$\mathbb{V}(\hat{\boldsymbol{\beta}}) = \sigma^2 (\mathbf{x}^\text{T} \mathbf{x})^{-1} = \sigma^2 \cdot \frac{\text{adj} (\mathbf{x}^\text{T} \mathbf{x})}{\text{det} (\mathbf{x}^\text{T} \mathbf{x})}.$$

The resultant correlation between the estimated intercept and the $k$th estimated slope parameter can be written in terms of the adjugate matrix of the Gramian of the design matrix, as follows:

$$\mathbb{Corr}(\hat{\beta}_0,\hat{\beta}_k) = \frac{[\text{adj} (\mathbf{x}^\text{T} \mathbf{x})]_{0,k}}{\sqrt{[\text{adj} (\mathbf{x}^\text{T} \mathbf{x})]_{0,0} \cdot [\text{adj} (\mathbf{x}^\text{T} \mathbf{x})]_{k,k}}}.$$

(Note that in this formula label the rows and columns as $k=0,1,...,m$, so the element $k$ refers to the $k$th slope coefficient.) This is a general formula for the correlation for a linear regression model with any number of coefficients. In the case where you have a large number of coefficients, the form of the adjugate matrix will be complicated, and will involve a substantial amount of interaction between the explanatory variables.


Correlation of the estimated coefficients in simple linear regression: For the case of simple linear regression this simplifies to:

$$\mathbb{V}(\hat{\boldsymbol{\beta}}) = \frac{1}{n(\sum x_i^2) - (\sum x_i)^2} \begin{bmatrix} \sum x_i^2 & - \sum x_i \\ - \sum x_i & n \\ \end{bmatrix},$$

and so, letting $\bar{x} = (\sum x_i)/n$ and $\bar{\bar{x}} = (\sum x_i^2)/n$, we have:

$$\mathbb{Corr}(\hat{\beta}_0,\hat{\beta}_1) = - \frac{\sum x_i}{\sqrt{n \sum x_i^2}} = - \frac{\bar{x}}{\sqrt{\bar{\bar{x}}}}.$$

Thus, in the case of simple linear regression, the correlation between the estimated slope and the estimated intercept has opposite sign to the mean of the explanatory variables. If the mean of the explanatory variables is positive, the correlation between the estimated coefficients will be negative. If the mean of the explanatory variables is negative, the correlation between the estimated coefficients will be positive. If the mean of the explanatory variables is zero, the correlation between the estimated coefficients will also be zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.