# How to write linear regression in matrix form for training/testing

Suppose I have a training set $$(x_1, y_1), \ldots, (x_n, y_n)$$, where $$x_i \in \mathbb{R}^p$$ for $$i = 1, \ldots, n$$ and I train an OLS model. My fitted values are $$\hat{y} = Hy$$, where $$H = X(X^TX)^{-1}X^T$$.

Now suppose I have a testing set $$\{x^*_1, \ldots, x^*_m\}$$, where $$x^*_i \in \mathbb{R}^p$$ for $$i = 1, \ldots, m$$. I want to make predictions on this testing set. My predicted values are

\begin{align*} \hat{y}^* &= X^* \hat{\beta}\\ &= X^*(X^TX)^{-1}X^Ty \end{align*}

Here, $$X^*(X^TX)^{-1}X$$ has dimension $$n \times m$$. It's not symmetric (or idempotent), so does this mean that the matrix $$X^*(X^TX)^{-1}X$$ is not a projection matrix?

If we suppose that $$n = m$$, so $$X^*(X^TX)^{-1}X$$ is of dimension $$n \times n$$. In that case, although it's a square matrix, it's still not a porjection matrix because it's not symmetric and idempotent?

• Right. Could you explain why it would be relevant for the matrix to be a projection matrix? – Christoph Hanck Jul 8 '20 at 7:30

## 1 Answer

Short answer

No, $$X^*(X^TX)^{-1}X^T$$ is not a projection matrix, even in the case $$m=n$$.

Longer answer

Throughout this answer we assume that $$X$$ has full column rank, which is equivalent to assuming that there is no multicollinearity among the exogenous variables, and that $$p.

The projection matrix $$X(X^TX)^{-1}X^T$$ is so called because it projects the vector $$y$$ on to the hyperplane spanned by the columns of $$X$$. Let's unpack that a little: $$X$$ has $$p$$ columns. Each of these columns is an $$n$$-dimensional vector, since $$X$$ has $$n$$ rows. These columns span a $$p$$-dimensional hyperplane in $$n$$-dimensional space. The matrix $$X(X^TX)^{-1}X^T$$ projects any $$n$$-dimensional vector on to this hyperplane.* The vector $$y$$ lives in $$n$$-dimensional space, so $$X(X^TX)^{-1}X^T$$ projects $$y$$ on to the hyperplane; the result of this projection is the vector $$\hat{y}$$. (Note that this projection minimises the length of the vector of residuals, i.e. $$\hat{y} = \mathrm{argmin}_\beta \lVert y - X\beta\rVert_2$$.)

This explains why the projection matrix is idempotent, since projecting a vector that is already on the hyperplane has no effect. As for symmetry: One can straightforwardly verify that the matrix $$X(X^TX)^{-1}X^T$$ is symmetric (transpose it!), although the geometric intuition isn't as straightforward.

Now, what about the matrix $$X^*$$? Each of the $$p$$ columns of $$X^*$$ is an $$m$$-dimensional vector, since $$X^*$$ has $$m$$ rows, but these vectors have nothing to do with the projection matrix $$X(X^TX)^{-1}X^T$$ and you should cast them from your mind! So what are we doing with $$X^*$$? Well, the vector $$\hat{y}$$ lies on the hyperplane spanned by the columns of $$X$$ and thus $$\hat{y}$$ is a linear combination of these columns. The vector $$\hat{\beta}$$ ($$= (X^TX)^{-1}X^Ty$$) contains the coefficients of this linear combination. We now move to $$p$$+1-dimensional space ($$p$$ columns in $$X$$ plus 1 column in $$y$$), which means that we switch to thinking about rows, rather than columns – I hope you're not still thinking about those $$m$$-dimensional vectors! The points $$(x_1,\hat{y}_1), \ldots, (x_n,\hat{y}_n)$$ all lie on a $$p$$-dimensional hyperplane – the fitted hyperplane – in $$p$$+1-dimensional space. The components of the $$m$$-dimensional vector $$X^*\hat{\beta}$$ tell you where the $$m$$ rows of $$X^*$$ are fitted to on the fitted hyperplane, i.e. the $$p$$+1-dimensional points $$(x_1^*,x_1^*\hat{\beta}), \ldots, (x_m^*,x_m^*\hat{\beta})$$ all lie on the fitted hyperplane.

*Proof: Suppose we want to project an $$n$$-dimensional vector $$z$$ on to the hyperplane spanned by the columns of $$X$$. Let $$\mathrm{proj}(z)$$ be this projection. What do we know about $$\mathrm{proj}(z)$$? Well, we know that it is in the span of the columns of $$X$$ and thus $$\mathrm{proj}(z) = X\alpha$$ for some $$p$$-dimensional vector $$\alpha$$. (The vector $$\alpha$$ contains the coefficients of the linear combination.) We also know that $$X^T(z - \mathrm{proj}(z)) = 0$$, since $$z - \mathrm{proj}(z)$$ is orthogonal to the hyperplane and so the dot products in the expression $$X^T(z- \mathrm{proj}(z))$$ are all 0. We can combine these two findings into one equation: $$X^T(z - X\alpha) = 0$$. Now, since $$X$$ has full column rank, the matrix $$X^TX$$ is invertible (see this answer). With this in hand, let's rearrange the equation:

\begin{align*} X^T(z - X\alpha) = 0 &\iff X^Tz - X^TX\alpha = 0\\ &\iff X^Tz = X^TX\alpha \\ &\iff (X^TX)^{-1}X^Tz = \alpha \end{align*}

We can then plug this value of $$\alpha$$ back into the equation $$\mathrm{proj}(z) = X\alpha$$ to get

$$\mathrm{proj}(z) = X(X^TX)^{-1}X^Tz.$$

So $$X(X^TX)^{-1}X^T$$ is indeed the matrix that sends vectors to their projections on the hyperplane spanned by the columns of $$X$$.

To get a hands-on sense of this, one can try the unpacking the expression $$X(X^TX)^{-1}X^T$$ for the case $$p=1$$, where the hyperplane is a line.