Consider the usual multivariate linear regression model where we solve for $\mathbf{\hat{b}^{OLS}}$

We have the equation


where $\mathbf{y}$ is the outcome matrix, $\mathbf{X}$ is the predictor matrix, $\mathbf{b}$ (not sure what to call it) and $\mathbf{u}$ is the error matrix. We assume $\mathbf{X}$ is of dimensions $n \times (k+1)$ where an extra row of $1$s is added to permit a $b_0$ term in each row. Therefore, $\mathbf{b}$ is of dimensions $(k+1) \times 1$

The OLS method solves

$$\underset{\mathbf{b}}\min \mathbf{u}^\text{T}\mathbf{u}$$ $$\text{s.t. }\mathbf{u} = \mathbf{y}-\mathbf{Xb}$$

The solution is


My Question

What is the intuition behind this statement of $\mathbf{\hat{b}^{OLS}}$?

Specifically, in the simple linear regression model, $\hat{b}^{OLS}$ was just the slope of the best fit line $\hat{Y} = \hat{b}_0+\hat{b}_1X$ or the intercept. But what about in the multivariate case?

  • 1
    $\begingroup$ I would guess this question has been asked before; have you checked? $\endgroup$ – Richard Hardy May 8 '16 at 6:23

The intuition is very simple. We try to project $Y$ onto the linear subspace spanned by $X_{1}\cdots X_{n}$. This is precisely the best $Y'$ we have such that $|Y-Y'|^2$ has the smallest value possible.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.