How to write linear regression in matrix form for training/testing

Question

Suppose I have a training set $(x_1, y_1), \ldots, (x_n, y_n)$, where $x_i \in \mathbb{R}^p$ for $i = 1, \ldots, n$ and I train an OLS model. My fitted values are $\hat{y} = Hy$, where $H = X(X^TX)^{-1}X^T$.

Now suppose I have a testing set $\{x^*_1, \ldots, x^*_m\}$, where $x^*_i \in \mathbb{R}^p$ for $i = 1, \ldots, m$. I want to make predictions on this testing set. My predicted values are

\begin{align*} \hat{y}^* &= X^* \hat{\beta}\\ &= X^*(X^TX)^{-1}X^Ty \end{align*}

Here, $X^*(X^TX)^{-1}X$ has dimension $n \times m$. It's not symmetric (or idempotent), so does this mean that the matrix $X^*(X^TX)^{-1}X$ is not a projection matrix?

If we suppose that $n = m$, so $X^*(X^TX)^{-1}X$ is of dimension $n \times n$. In that case, although it's a square matrix, it's still not a porjection matrix because it's not symmetric and idempotent?

Answer 1

Short answer

No, $X^*(X^TX)^{-1}X^T$ is not a projection matrix, even in the case $m=n$.

Longer answer

Throughout this answer we assume that $X$ has full column rank, which is equivalent to assuming that there is no multicollinearity among the exogenous variables, and that $p<n$.

The projection matrix $X(X^TX)^{-1}X^T$ is so called because it projects the vector $y$ on to the hyperplane spanned by the columns of $X$. Let's unpack that a little: $X$ has $p$ columns. Each of these columns is an $n$-dimensional vector, since $X$ has $n$ rows. These columns span a $p$-dimensional hyperplane in $n$-dimensional space. The matrix $X(X^TX)^{-1}X^T$ projects any $n$-dimensional vector on to this hyperplane.* The vector $y$ lives in $n$-dimensional space, so $X(X^TX)^{-1}X^T$ projects $y$ on to the hyperplane; the result of this projection is the vector $\hat{y}$. (Note that this projection minimises the length of the vector of residuals, i.e. $\hat{y} = \mathrm{argmin}_\beta \lVert y - X\beta\rVert_2$.)

This explains why the projection matrix is idempotent, since projecting a vector that is already on the hyperplane has no effect. As for symmetry: One can straightforwardly verify that the matrix $X(X^TX)^{-1}X^T$ is symmetric (transpose it!), although the geometric intuition isn't as straightforward.

Now, what about the matrix $X^*$? Each of the $p$ columns of $X^*$ is an $m$-dimensional vector, since $X^*$ has $m$ rows, but these vectors have nothing to do with the projection matrix $X(X^TX)^{-1}X^T$ and you should cast them from your mind! So what are we doing with $X^*$? Well, the vector $\hat{y}$ lies on the hyperplane spanned by the columns of $X$ and thus $\hat{y}$ is a linear combination of these columns. The vector $\hat{\beta}$ ($= (X^TX)^{-1}X^Ty$) contains the coefficients of this linear combination. We now move to $p$+1-dimensional space ($p$ columns in $X$ plus 1 column in $y$), which means that we switch to thinking about rows, rather than columns – I hope you're not still thinking about those $m$-dimensional vectors! The points $(x_1,\hat{y}_1), \ldots, (x_n,\hat{y}_n)$ all lie on a $p$-dimensional hyperplane – the fitted hyperplane – in $p$+1-dimensional space. The components of the $m$-dimensional vector $X^*\hat{\beta}$ tell you where the $m$ rows of $X^*$ are fitted to on the fitted hyperplane, i.e. the $p$+1-dimensional points $(x_1^*,x_1^*\hat{\beta}), \ldots, (x_m^*,x_m^*\hat{\beta})$ all lie on the fitted hyperplane.

---

*Proof: Suppose we want to project an $n$-dimensional vector $z$ on to the hyperplane spanned by the columns of $X$. Let $\mathrm{proj}(z)$ be this projection. What do we know about $\mathrm{proj}(z)$? Well, we know that it is in the span of the columns of $X$ and thus $\mathrm{proj}(z) = X\alpha$ for some $p$-dimensional vector $\alpha$. (The vector $\alpha$ contains the coefficients of the linear combination.) We also know that $X^T(z - \mathrm{proj}(z)) = 0$, since $z - \mathrm{proj}(z)$ is orthogonal to the hyperplane and so the dot products in the expression $X^T(z- \mathrm{proj}(z))$ are all 0. We can combine these two findings into one equation: $X^T(z - X\alpha) = 0$. Now, since $X$ has full column rank, the matrix $X^TX$ is invertible (see this answer). With this in hand, let's rearrange the equation:

\begin{align*} X^T(z - X\alpha) = 0 &\iff X^Tz - X^TX\alpha = 0\\ &\iff X^Tz = X^TX\alpha \\ &\iff (X^TX)^{-1}X^Tz = \alpha \end{align*}

We can then plug this value of $\alpha$ back into the equation $\mathrm{proj}(z) = X\alpha$ to get

$$ \mathrm{proj}(z) = X(X^TX)^{-1}X^Tz. $$

So $X(X^TX)^{-1}X^T$ is indeed the matrix that sends vectors to their projections on the hyperplane spanned by the columns of $X$.

To get a hands-on sense of this, one can try the unpacking the expression $X(X^TX)^{-1}X^T$ for the case $p=1$, where the hyperplane is a line.