# Least Squares Definition in Elements of Statistical Learning

In Elements of Statistical Learning, they state on p. 11 that all vectors are column vectors and start developing the least squares idea.

So if we have $$\mathbf{X} = \begin{bmatrix} 1 \\ X_1 \\ X_2 \\ \vdots \\ X_p\end{bmatrix}$$ and $$\hat{\boldsymbol{\beta}} = \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \\ \vdots \\ \hat{\beta}_p \end{bmatrix}\text{,}$$ take $$\hat{Y} = \mathbf{X}^{T}\hat{\boldsymbol{\beta}} = \langle \mathbf{X}, \hat{\boldsymbol{\beta}} \rangle\text{.}$$ So $\hat{Y} \in \mathbb{R}^1$ because $\mathbf{X}^{T} \in M_{1 \times p}(\mathbb{R})$ and $\boldsymbol{\beta} \in M_{p \times 1}(\mathbb{R})$. Fine.

Now they say:

... in general $\hat{Y}$ can be a $K$-vector, in which case $\beta$ (I think this is a typo - they should probably have $\hat{\beta}$ instead) would be a $p \times K$ matrix of coefficients.

Well, here's the problem. If $\mathbf{X}^{T} \in M_{1 \times p}(\mathbb{R})$ and $\hat{\beta} \in M_{p \times K}(\mathbb{R})$, wouldn't $\mathbf{X}^{T}\hat{\beta}$ give a row vector for $\mathbf{Y}$ (dimensions $1 \times K$)?

• Your $X$ makes no sense in this context unless its elements are interpreted as row vectors of dimension $K$. Both $\beta$ and its estimate $\hat\beta$ are column vectors of dimension $p+1$, as you show. Have you perhaps mixed up the roles of $X$ and $\beta$ in the model? – whuber Nov 13 '15 at 18:20

"a set of $N$ input $p$-vectors $x_i$ , $i = 1, \dots, N$ would be represented by the $N \times p$ matrix $X$."
This means that the $p$ feature vectors/regressors they use will be represented as column vectors in the matrix $X$. For that matter and to quote from the text page 12 exactly :
"$X$ is an $N \times p$ matrix with each row an input vector".
So $X^T$ is not a row (or column) vector in any case. $X$ (and its transpose) will be (probably) of rank $p$. $\hat{Y}$ will be a matrix $N \times K$. For the case of OLS model $K$ is usually equal to unity (1) but this is not a strict prerequisite for its use (that's why usually we write $\hat{y}$).