9
$\begingroup$

In http://www.wouterdenhaan.com/numerical/slidesbayesian.pdf (approximately from page 7 to 13), ordinary least squares and linear projection are said to be different. But from my linear algebra class, I remember hearing that OLS is indeed projection method. So I am confused here.

What exactly is the difference between these two?

$\endgroup$
1
  • 1
    $\begingroup$ linear projection refers to the population quantities, while OLS looks at sample quantities $\endgroup$ – Christoph Hanck Jun 10 '15 at 3:41
14
$\begingroup$

OLS, conditional expectation and linear projection are all related. It helps to distinguish between the unknown data generating process (the model) and procedures to estimate the parameters of that model.

Let this be model/data generating process. $f$ is some unknown function.

$y_i = f(x_i, \theta) +\epsilon_i$, $E[x_i\epsilon]=0$

We could use OLS, and regress $y_i$ on vector $x_i$. The OLS estimator is defined to be the vector $b$ that minimises the sample sum of squares $(y-Xb)^T(y-Xb)$ ( $y$ is $n \times 1$, $X$ is $n \times k$ ).

As the sample size $n$ gets larger, $b$ will converge to something (in probability). Whether it converges to $\beta$, though, depends on what the true model/dgp actually is, ie on $f$.

Suppose $f$ really is linear. Then $y_i = x_i^T\theta +\epsilon_i$ and $E[y_i|x_i]=x_i^T\theta$ and $b$ converges to $\theta$.

What if $f$ isn't linear? $b$ still converges to something, the thing it always converges to: the linear projection coefficient. What is a linear projection? Is is the population equivalent of the OLS estimator. The vector $\beta$ that minimises $E[ (y_i-x_i^T\beta)^T (y_i-x_i^T\beta)]$. Regardless of what the true relation between y and x is, this vector exists and OLS converges to it.

In the special case where the conditional expectation is linear, $\theta$ and $\beta$ are the same, and OLS recovers the conditional expectation function for you as the sample grows. If that function is not linear, OLS recovers just the linear projection coefficient for you, which could still be useful, because it is the mean square error minimising linear approximation of the conditional expectation function.

$\endgroup$
5
$\begingroup$

The Y-hats are obtained by projecting Y onto the column space of X:

\begin{equation*} \hat{\beta} = \left(X^\prime X \right)^{-1} \, X^\prime Y \end{equation*} \begin{equation*} \hat{Y} = X \hat{\beta} = X\,\left(X^\prime X \right)^{-1} \, X^\prime Y = P\,Y \end{equation*} where \begin{equation*} P \equiv X\,\left(X^\prime X \right)^{-1} \, X^\prime \end{equation*} is a projection matrix in the linear algebra sense.

I'm using $\hat{\beta}$ as the estimated coefficient vector in OLS, and $\hat{Y}$ as the fitted values.

$\endgroup$
1
  • 2
    $\begingroup$ Could you clarify the connection between your projection matrix and the definition of "linear projection" in the link provided in the question? That reference states, "The linear projection of $y$ on $x$ is the function $\hat{E}[y|x]=a+Bx$ [where] $a$ and $B$ are chosen to minimize $\mathbb{E}(tr((y-a-Bx)(y-a-Bx)^\prime))$." (I fixed some sign errors in that formula.) It goes on to give a formula "$\hat{E}[y|x]=\mu_y+\Sigma_{yx}\Sigma^{-1}_{xx}(x-\mu_x)$." (The $\mu$ are means and the $\Sigma$ are covariances.) Note that this reference asserts your OLS solution is not linear projection. $\endgroup$ – whuber Jun 10 '15 at 18:31
0
$\begingroup$

As an addition to @CloseToC's answer: Even if the DGS f is linear, there's a significant difference between linear projection and OLS. OLS requires that the error term, $\epsilon_i$, is uncorrelated with each of the independent variables. It guarantees the consistency of coefficient estimates. However, linear projection does not require that. Even if this assumption fails, i.e., some independent variables are endogenous, linear projection still exists (guaranteed by the Projection Theorem). By the way, one way to fix the OLS, in this case, is to use an instrumental variable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.