So in a simple linear regression
Y = B0 + B1X
B0 and B1 are negatively correlated. When slope goes up, intercept must go down, in order for the regression line to still pass through as much of the data as possible.
My question is, does the same intuition apply for a multiple linear regression assuming all the predictors are independent from each other. Would each slope coefficient, Bi, individually be negatively correlated with intercept, B0?
Thanks!
The question asks if estimated slopes must be correlated with the estimate of the intercept. The answer is no, not necessarily. The slope will not be correlated with the intercept if the sample mean of $X$ is $0$. Below is a simple simulation, coded in R:
set.seed(1145) # this makes the example exactly reproducible
x = runif(n=50, min=-4, max=+4)
x = x - mean(x)
mean(x) # [1] 3.680216e-17
xr = x + 5
xl = x - 5
z = 0 + 1*x
out.mat = matrix(NA, nrow=100, ncol=6)
colnames(out.mat) = c("centered int", "centered slope",
"right int", "right slope",
"left int", "left slope")
for(i in 1:100){
e = rnorm(50, mean=0, sd=1)
y = z + e
mc = lm(y~x)
mr = lm(y~xr)
ml = lm(y~xl)
out.mat[i,1:2] = coef(mc)
out.mat[i,3:4] = coef(mr)
out.mat[i,5:6] = coef(ml)
}
cor(out.mat[,1], out.mat[,2]) # [1] -0.03190039
cor(out.mat[,3], out.mat[,4]) # [1] -0.9135444
cor(out.mat[,5], out.mat[,6]) # [1] 0.9090916
windows(width=9, height=3.5)
layout(matrix(1:3, nrow=1))
plot(out.mat[,1], out.mat[,2], xlab="int", ylab="slope", main="Centered")
plot(out.mat[,3], out.mat[,4], xlab="int", ylab="slope", main="Positive")
plot(out.mat[,5], out.mat[,6], xlab="int", ylab="slope", main="Negative")
(The original answer, below, pertains to the original phrasing of the question, which specifically asked if a negative correlation between slopes and the intercept was necessary.)
It depends on whether your data are to the right of $0$ on $X$ or to the left (i.e., your $X$ data are all negative). That is less common, it would seem, but there is nothing mathematically necessary about it. If your $X$ data were negative, then there would be a positive correlation between $\hat\beta_0$ and $\hat\beta_1$.
set.seed(1145) # this makes the example exactly reproducible
x = runif(n=50, min=-4, max=+4)
y = 0 + 1*x + rnorm(50, mean=0, sd=1)
coef(summary(lm(y~x)))
# Estimate Std. Error t value Pr(>|t|)
# (Intercept) 0.05130084 0.12468707 0.4114367 6.825837e-01
# x 0.92041320 0.05673818 16.2221146 4.094011e-21
xr = x + 5 # w/ pos slope, moving x to the right will make the intercept go down
xl = x - 5 # w/ pos slope, moving x to the left will make the intercept go up
coef(summary(lm(y~xr)))
# Estimate Std. Error t value Pr(>|t|)
# (Intercept) -4.5507652 0.32128386 -14.16431 9.123907e-19
# xr 0.9204132 0.05673818 16.22211 4.094011e-21
coef(summary(lm(y~xl)))
# Estimate Std. Error t value Pr(>|t|)
# (Intercept) 4.6533668 0.29804604 15.61291 1.929555e-20
# xl 0.9204132 0.05673818 16.22211 4.094011e-21
windows()
layout(matrix(c(1,2,3), nrow=3, byrow=TRUE))
plot(x, y, xlim=c(-9, 9), ylim=c(-4,4))
abline(.05, .92, col="red")
abline(v=0, col="gray"); abline(h=0, col="gray")
plot(xl, y, xlim=c(-9, 9), ylim=c(-4,4))
abline(4.65, .92, col="red")
abline(v=0, col="gray"); abline(h=0, col="gray")
plot(xr, y, xlim=c(-9, 9), ylim=c(-4,4))
abline(-4.55, .92, col="red")
abline(v=0, col="gray"); abline(h=0, col="gray")
Correlation of the estimated coefficients in linear regression: For linear regression using OLS estimation of the coefficients, the general formula for the variance of the estimated coefficient vector (implicitly conditional on the explanatory variables) is:
$$\mathbb{V}(\hat{\boldsymbol{\beta}}) = \sigma^2 (\mathbf{x}^\text{T} \mathbf{x})^{-1} = \sigma^2 \cdot \frac{\text{adj} (\mathbf{x}^\text{T} \mathbf{x})}{\text{det} (\mathbf{x}^\text{T} \mathbf{x})}.$$
The resultant correlation between the estimated intercept and the $k$th estimated slope parameter can be written in terms of the adjugate matrix of the Gramian of the design matrix, as follows:
$$\mathbb{Corr}(\hat{\beta}_0,\hat{\beta}_k) = \frac{[\text{adj} (\mathbf{x}^\text{T} \mathbf{x})]_{0,k}}{\sqrt{[\text{adj} (\mathbf{x}^\text{T} \mathbf{x})]_{0,0} \cdot [\text{adj} (\mathbf{x}^\text{T} \mathbf{x})]_{k,k}}}.$$
(Note that in this formula label the rows and columns as $k=0,1,...,m$, so the element $k$ refers to the $k$th slope coefficient.) This is a general formula for the correlation for a linear regression model with any number of coefficients. In the case where you have a large number of coefficients, the form of the adjugate matrix will be complicated, and will involve a substantial amount of interaction between the explanatory variables.
Correlation of the estimated coefficients in simple linear regression: For the case of simple linear regression this simplifies to:
$$\mathbb{V}(\hat{\boldsymbol{\beta}}) = \frac{1}{n(\sum x_i^2) - (\sum x_i)^2} \begin{bmatrix} \sum x_i^2 & - \sum x_i \\ - \sum x_i & n \\ \end{bmatrix},$$
and so, letting $\bar{x} = (\sum x_i)/n$ and $\bar{\bar{x}} = (\sum x_i^2)/n$, we have:
$$\mathbb{Corr}(\hat{\beta}_0,\hat{\beta}_1) = - \frac{\sum x_i}{\sqrt{n \sum x_i^2}} = - \frac{\bar{x}}{\sqrt{\bar{\bar{x}}}}.$$
Thus, in the case of simple linear regression, the correlation between the estimated slope and the estimated intercept has opposite sign to the mean of the explanatory variables. If the mean of the explanatory variables is positive, the correlation between the estimated coefficients will be negative. If the mean of the explanatory variables is negative, the correlation between the estimated coefficients will be positive. If the mean of the explanatory variables is zero, the correlation between the estimated coefficients will also be zero.
