# Uniqueness of $x'\beta$ even when $\mathbb{E}(x^Tx)$ is not invertible

As discussed in user25658's answer to this question, when one wants to compute

$$\beta = \mathbb{E}(x^Tx)^{-1} \mathbb{E}(x^TY)$$

but $\mathbb{E}(x^Tx)$ is not invertible, $\beta$ is not uniquely identified. A characterization of all possible values of $\beta$ is given by

$$\beta= \mathbb{E}(x^Tx)^g\mathbb{E}(x^Ty)$$

where $\mathbb{E}(x^Tx)^g$ is the generalized inverse of $\mathbb{E}(x^Tx)$.

In Hansen, Econometrics, pg 34, section 2.18 (as of today's version on Hansen's webiste) it is written that $x^T\beta = x^T \mathbb{E}(x^Tx)^g\mathbb{E}(x^Ty)$ is however uniquely identified.

This result is not completely counterintuitive to me. I understand that that this is matrix multiplication and that the fact that for every $x$ there may be multiple $\beta(x)$, say $\beta(x)_1 \neq \beta(x)_2$, does not imply $x^T\beta(x)_1 \neq x^T\beta(x)_2$.

Nevertheless, I have been unsuccessful in trying to prove the claim. Could someone help me with that?

• I'm not seeing the Hansen text on either definition of page 32 (text page labeled "32" or pdf page 32), just FYI. – jbowman Jan 31 '14 at 21:05
• My bad, the online version as been edited since I downloaded it. pg. 34 on the current edition on the website. I edited my post, thanks. – Martin Van der Linden Jan 31 '14 at 21:14
• When $x$ is fixed (and not random), $x'\beta$ is the projection of $y$ onto the space generated by the columns of $x$. This geometric characterization makes the claim obvious in this situation. Using statistical intuition, we can also characterize $x'\beta$ as the fit to $y$. Even when the problem is overparameterized (i.e., $x$ has deficient rank), there is no problem because the extra parameters don't change the fit. I cannot see a way easily to generalize to random $x$, partly because there is no inherent relationship among the ranks of $x$ and $\mathbb{E}(x'x)$. – whuber Jan 31 '14 at 22:15

## 1 Answer

Note that when $E [xx^T] = A$ is not invertible you actually want a solution to the equation

$$A\beta=E [xx^T]\beta=E [xy]=c$$

Now if a solution exists it must have the form

$$\hat {\beta}=A^{+}c + (I-A^{+}A)w$$

Where $w$ is an arbitrary vector with the same dimension as $c$, and $A^{+}$ is the moore penrose pseudo inverse - can be calculated by using the spectral decomposition of $A$. Now what happens to a predicted value? This is given as:

$$x^T \hat {\beta}=x^TA^{+}c + (x^T-x^TA^{+}A)w$$

Now using the spectral decomposition we have

$$A=FLF^T\implies A^{+}=FL^{+} F^T$$

This gives us

$$x^T\hat {\beta}=x^TFL^{+} F^Tc + x^TF(I-LL^{+} )F^Tw$$

Now this is basically doing a co-ordinate transform by the matrix $F$. $x^{*}=F^Tx$ and $c^{*}=F^Tc$ and $w^{*}=F^Tw$. Now we will basically have that

$$x^T\hat {\beta}=\sum_{k=1}^{r}\frac {x^{*}_{k} c^{*}_{k}}{l_k} +\sum_{k=r+1}^{p}x^{*}_{k} w^{*}_{k}$$

Where $r$ is the rank of $A=E [xx^T]$ and $p\geq r$ is the dimension of $x$. So the arbitrariness of the predictions is "siphoned" off into a $p-r$ subspace.

To remove it completely we need to plug in the "sample" version of A, namely $$A_s=\frac {1}{n}\sum_{I=1}^{n} x_ix_i^T$$ and then we note that $x_i^{*}$ is just the "principal component" score for the ith observation in the sample. This means that $x_{ik}^{*}=0$ for $k> r$. So this means that the arbitrary choice for $w$ only affects the betas, but not the fitted values.

Note that this is not true for "future" predictions, for $x$ values that weren't observed in the sample.

Hope this helps.

• Is $E$ really a good name for an orthogonal matrix in this context? – Roland Feb 1 '14 at 10:28
• @Roland - Good point. Change made – probabilityislogic Feb 1 '14 at 14:10
• Thanks, very helpful. A proof of the form the solution β^ must take (if existing) can be found at math.wustl.edu/~sawyer/handouts/GenrlInv.pdf. Watching Gilbert Strang's lecture on generalized inverses also helped me a lot in understanding your answer videolectures.net/mit1806s05_strang_lec33. – Martin Van der Linden Feb 5 '14 at 18:49