Least Squares Definition in Elements of Statistical Learning

Question

In Elements of Statistical Learning, they state on p. 11 that all vectors are column vectors and start developing the least squares idea.

So if we have $$\mathbf{X} = \begin{bmatrix} 1 \\ X_1 \\ X_2 \\ \vdots \\ X_p\end{bmatrix}$$ and $$\hat{\boldsymbol{\beta}} = \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \\ \vdots \\ \hat{\beta}_p \end{bmatrix}\text{,}$$ take $$\hat{Y} = \mathbf{X}^{T}\hat{\boldsymbol{\beta}} = \langle \mathbf{X}, \hat{\boldsymbol{\beta}} \rangle\text{.}$$ So $\hat{Y} \in \mathbb{R}^1$ because $\mathbf{X}^{T} \in M_{1 \times p}(\mathbb{R})$ and $\boldsymbol{\beta} \in M_{p \times 1}(\mathbb{R})$. Fine.

Now they say:

... in general $\hat{Y}$ can be a $K$-vector, in which case $\beta$ (I think this is a typo - they should probably have $\hat{\beta}$ instead) would be a $p \times K$ matrix of coefficients.

Well, here's the problem. If $\mathbf{X}^{T} \in M_{1 \times p}(\mathbb{R})$ and $\hat{\beta} \in M_{p \times K}(\mathbb{R})$, wouldn't $\mathbf{X}^{T}\hat{\beta}$ give a row vector for $\mathbf{Y}$ (dimensions $1 \times K$)?

Answer 1

I think you confuse yourself a bit. In page 10 the authors say:

"a set of $N$ input $p$-vectors $x_i$ , $i = 1, \dots, N$ would be represented by the $N \times p$ matrix $X$."

This means that the $p$ feature vectors/regressors they use will be represented as column vectors in the matrix $X$. For that matter and to quote from the text page 12 exactly :

"$X$ is an $N \times p$ matrix with each row an input vector".

So $X^T$ is not a row (or column) vector in any case. $X$ (and its transpose) will be (probably) of rank $p$. $\hat{Y}$ will be a matrix $N \times K$. For the case of OLS model $K$ is usually equal to unity (1) but this is not a strict prerequisite for its use (that's why usually we write $\hat{y}$).