Speeding up hat matrices like $X(X'X)^{-1}X'$ (projection matrices) and other aspects of custom-built estimator when software runs out of memory Is there a way to speed up $Z(Z'Z)^{-1}Z'$ type matrices? I am implementing the expression below directly using a matrix language and my program frequently crashes while if I run OLS on them using a pre-fabricated command, it is not an issue.
Is there a tip you guys might have to compute these matrices efficiently?
The goal here (but that is just an aside) is to implement the following estimator
\begin{eqnarray}
(X' P X - \sum_{i=1}^{n} P_{ii} X_{i}X_{i}' - \alpha X'X)^{-1} (X' P_Z y - \sum_{i=1}^{n} P_{ii} X_{i} y_{i} - \alpha X' y)
\end{eqnarray}
Now, $\alpha$ is the smallest eigenvalue of $(\overline{X}'\overline{X})^{-1} (\overline{X}' P_Z \overline{X} - \sum_{i=1}^{n} P_{ii} \overline{X}_{i} \overline{X}'_{i})$ where $\overline{X} = [y,X]$.
I am pretty confident that once I have found an efficient way to compute the projections, I can easily implement the rest easily.
Thanks so much!
 A: Using QR decomposition (which ought to be available if you already have calculated the regression):
Let $X$ have $n$ rows and $p$ columns and be of full column rank.
$H=X(X'X)^{-1}X'$
$=QR(R'Q'QR)^{-1}R'Q'$
$=QR(R'R)^{-1}R'Q'$
But if $R_1$ is the first $p$ rows of $R$ then $R'R=R_1'R_1$
$=QR(R_1'R_1)^{-1}R'Q'$
Now let $Q=(Q_1,Q_2)$ where $Q_1$ is the first $p$ columns of $Q$. Then $QR=Q_1R_1$.
$=Q_1R_1R_1^{-1}(R_1')^{-1}R_1'Q_1'$
$=Q_1Q_1'$
Where $Q_1$ is $n\times p$.
So if you have the QR decomposition of $X$, then the hat matrix is fairly simple.
Note that some regression programs will give $Q_1$ automatically. [It's also possibly that a regression program will have performed pivoting. That shouldn't impact the calculation of the hat matrix though.]
A: Try using the SVD. E.g., since
$$
    X = U\Sigma V^T,
$$
then
$$
    (X^T X)^{-1} = (V\Sigma^2 V^T)^{-1} = V\Sigma^{-2} V^T,
$$
and thus
$$
    X(X^T X)^{-1}X^T = U I_r U^T = U_r U_r^T,
$$
where $I_r$ is an $n\times n$ identity matrix with $r\leq n$ ones on the diagonal (upper part), and $n-r$ zeros on the lower diagonal, where $r$ is the rank of $X$.
This will likely speed up your computation.
