Suppose I have a training set $(x_1, y_1), \ldots, (x_n, y_n)$, where $x_i \in \mathbb{R}^p$ for $i = 1, \ldots, n$ and I train an OLS model. My fitted values are $\hat{y} = Hy$, where $H = X(X^TX)^{-1}X^T$.

Now suppose I have a testing set $\{x^*_1, \ldots, x^*_m\}$, where $x^*_i \in \mathbb{R}^p$ for $i = 1, \ldots, m$. I want to make predictions on this testing set. My predicted values are

\begin{align*} \hat{y}^* &= X^* \hat{\beta}\\ &= X^*(X^TX)^{-1}X^Ty \end{align*}

Here, $X^*(X^TX)^{-1}X$ has dimension $n \times m$. It's not symmetric (or idempotent), so does this mean that the matrix $X^*(X^TX)^{-1}X$ is not a projection matrix?

If we suppose that $n = m$, so $X^*(X^TX)^{-1}X$ is of dimension $n \times n$. In that case, although it's a square matrix, it's still not a porjection matrix because it's not symmetric and idempotent?

  • 2
    $\begingroup$ Right. Could you explain why it would be relevant for the matrix to be a projection matrix? $\endgroup$ – Christoph Hanck Jul 8 '20 at 7:30

Short answer

No, $X^*(X^TX)^{-1}X^T$ is not a projection matrix, even in the case $m=n$.

Longer answer

Throughout this answer we assume that $X$ has full column rank, which is equivalent to assuming that there is no multicollinearity among the exogenous variables, and that $p<n$.

The projection matrix $X(X^TX)^{-1}X^T$ is so called because it projects the vector $y$ on to the hyperplane spanned by the columns of $X$. Let's unpack that a little: $X$ has $p$ columns. Each of these columns is an $n$-dimensional vector, since $X$ has $n$ rows. These columns span a $p$-dimensional hyperplane in $n$-dimensional space. The matrix $X(X^TX)^{-1}X^T$ projects any $n$-dimensional vector on to this hyperplane.* The vector $y$ lives in $n$-dimensional space, so $X(X^TX)^{-1}X^T$ projects $y$ on to the hyperplane; the result of this projection is the vector $\hat{y}$. (Note that this projection minimises the length of the vector of residuals, i.e. $\hat{y} = \mathrm{argmin}_\beta \lVert y - X\beta\rVert_2$.)

This explains why the projection matrix is idempotent, since projecting a vector that is already on the hyperplane has no effect. As for symmetry: One can straightforwardly verify that the matrix $X(X^TX)^{-1}X^T$ is symmetric (transpose it!), although the geometric intuition isn't as straightforward.

Now, what about the matrix $X^*$? Each of the $p$ columns of $X^*$ is an $m$-dimensional vector, since $X^*$ has $m$ rows, but these vectors have nothing to do with the projection matrix $X(X^TX)^{-1}X^T$ and you should cast them from your mind! So what are we doing with $X^*$? Well, the vector $\hat{y}$ lies on the hyperplane spanned by the columns of $X$ and thus $\hat{y}$ is a linear combination of these columns. The vector $\hat{\beta}$ ($= (X^TX)^{-1}X^Ty$) contains the coefficients of this linear combination. We now move to $p$+1-dimensional space ($p$ columns in $X$ plus 1 column in $y$), which means that we switch to thinking about rows, rather than columns – I hope you're not still thinking about those $m$-dimensional vectors! The points $(x_1,\hat{y}_1), \ldots, (x_n,\hat{y}_n)$ all lie on a $p$-dimensional hyperplane – the fitted hyperplane – in $p$+1-dimensional space. The components of the $m$-dimensional vector $X^*\hat{\beta}$ tell you where the $m$ rows of $X^*$ are fitted to on the fitted hyperplane, i.e. the $p$+1-dimensional points $(x_1^*,x_1^*\hat{\beta}), \ldots, (x_m^*,x_m^*\hat{\beta})$ all lie on the fitted hyperplane.

*Proof: Suppose we want to project an $n$-dimensional vector $z$ on to the hyperplane spanned by the columns of $X$. Let $\mathrm{proj}(z)$ be this projection. What do we know about $\mathrm{proj}(z)$? Well, we know that it is in the span of the columns of $X$ and thus $\mathrm{proj}(z) = X\alpha$ for some $p$-dimensional vector $\alpha$. (The vector $\alpha$ contains the coefficients of the linear combination.) We also know that $X^T(z - \mathrm{proj}(z)) = 0$, since $z - \mathrm{proj}(z)$ is orthogonal to the hyperplane and so the dot products in the expression $X^T(z- \mathrm{proj}(z))$ are all 0. We can combine these two findings into one equation: $X^T(z - X\alpha) = 0$. Now, since $X$ has full column rank, the matrix $X^TX$ is invertible (see this answer). With this in hand, let's rearrange the equation:

\begin{align*} X^T(z - X\alpha) = 0 &\iff X^Tz - X^TX\alpha = 0\\ &\iff X^Tz = X^TX\alpha \\ &\iff (X^TX)^{-1}X^Tz = \alpha \end{align*}

We can then plug this value of $\alpha$ back into the equation $\mathrm{proj}(z) = X\alpha$ to get

$$ \mathrm{proj}(z) = X(X^TX)^{-1}X^Tz. $$

So $X(X^TX)^{-1}X^T$ is indeed the matrix that sends vectors to their projections on the hyperplane spanned by the columns of $X$.

To get a hands-on sense of this, one can try the unpacking the expression $X(X^TX)^{-1}X^T$ for the case $p=1$, where the hyperplane is a line.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.