4
$\begingroup$

As discussed in user25658's answer to this question, when one wants to compute

$$ \beta = \mathbb{E}(x^Tx)^{-1} \mathbb{E}(x^TY) $$

but $\mathbb{E}(x^Tx)$ is not invertible, $\beta$ is not uniquely identified. A characterization of all possible values of $\beta$ is given by

$$\beta= \mathbb{E}(x^Tx)^g\mathbb{E}(x^Ty)$$

where $\mathbb{E}(x^Tx)^g$ is the generalized inverse of $\mathbb{E}(x^Tx)$.

In Hansen, Econometrics, pg 34, section 2.18 (as of today's version on Hansen's webiste) it is written that $x^T\beta = x^T \mathbb{E}(x^Tx)^g\mathbb{E}(x^Ty) $ is however uniquely identified.

This result is not completely counterintuitive to me. I understand that that this is matrix multiplication and that the fact that for every $x$ there may be multiple $\beta(x)$, say $\beta(x)_1 \neq \beta(x)_2$, does not imply $x^T\beta(x)_1 \neq x^T\beta(x)_2$.

Nevertheless, I have been unsuccessful in trying to prove the claim. Could someone help me with that?

$\endgroup$
  • 1
    $\begingroup$ I'm not seeing the Hansen text on either definition of page 32 (text page labeled "32" or pdf page 32), just FYI. $\endgroup$ – jbowman Jan 31 '14 at 21:05
  • $\begingroup$ My bad, the online version as been edited since I downloaded it. pg. 34 on the current edition on the website. I edited my post, thanks. $\endgroup$ – Martin Van der Linden Jan 31 '14 at 21:14
  • 2
    $\begingroup$ When $x$ is fixed (and not random), $x'\beta$ is the projection of $y$ onto the space generated by the columns of $x$. This geometric characterization makes the claim obvious in this situation. Using statistical intuition, we can also characterize $x'\beta$ as the fit to $y$. Even when the problem is overparameterized (i.e., $x$ has deficient rank), there is no problem because the extra parameters don't change the fit. I cannot see a way easily to generalize to random $x$, partly because there is no inherent relationship among the ranks of $x$ and $\mathbb{E}(x'x)$. $\endgroup$ – whuber Jan 31 '14 at 22:15
3
$\begingroup$

Note that when $ E [xx^T] = A$ is not invertible you actually want a solution to the equation

$$ A\beta=E [xx^T]\beta=E [xy]=c$$

Now if a solution exists it must have the form

$$ \hat {\beta}=A^{+}c + (I-A^{+}A)w $$

Where $ w $ is an arbitrary vector with the same dimension as $ c $, and $ A^{+} $ is the moore penrose pseudo inverse - can be calculated by using the spectral decomposition of $ A $. Now what happens to a predicted value? This is given as:

$$ x^T \hat {\beta}=x^TA^{+}c + (x^T-x^TA^{+}A)w $$

Now using the spectral decomposition we have

$$ A=FLF^T\implies A^{+}=FL^{+} F^T $$

This gives us

$$x^T\hat {\beta}=x^TFL^{+} F^Tc + x^TF(I-LL^{+} )F^Tw$$

Now this is basically doing a co-ordinate transform by the matrix $ F $. $ x^{*}=F^Tx $ and $ c^{*}=F^Tc$ and $ w^{*}=F^Tw $. Now we will basically have that

$$x^T\hat {\beta}=\sum_{k=1}^{r}\frac {x^{*}_{k} c^{*}_{k}}{l_k} +\sum_{k=r+1}^{p}x^{*}_{k} w^{*}_{k}$$

Where $ r $ is the rank of $ A=E [xx^T] $ and $ p\geq r $ is the dimension of $ x $. So the arbitrariness of the predictions is "siphoned" off into a $ p-r $ subspace.

To remove it completely we need to plug in the "sample" version of A, namely $$ A_s=\frac {1}{n}\sum_{I=1}^{n} x_ix_i^T $$ and then we note that $ x_i^{*} $ is just the "principal component" score for the ith observation in the sample. This means that $ x_{ik}^{*}=0 $ for $ k> r $. So this means that the arbitrary choice for $ w $ only affects the betas, but not the fitted values.

Note that this is not true for "future" predictions, for $ x $ values that weren't observed in the sample.

Hope this helps.

$\endgroup$
  • $\begingroup$ Is $E$ really a good name for an orthogonal matrix in this context? $\endgroup$ – Roland Feb 1 '14 at 10:28
  • $\begingroup$ @Roland - Good point. Change made $\endgroup$ – probabilityislogic Feb 1 '14 at 14:10
  • $\begingroup$ Thanks, very helpful. A proof of the form the solution β^ must take (if existing) can be found at math.wustl.edu/~sawyer/handouts/GenrlInv.pdf. Watching Gilbert Strang's lecture on generalized inverses also helped me a lot in understanding your answer videolectures.net/mit1806s05_strang_lec33. $\endgroup$ – Martin Van der Linden Feb 5 '14 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.