In Elements of Statistical Learning, they state on p. 11 that all vectors are column vectors and start developing the least squares idea.

So if we have $$\mathbf{X} = \begin{bmatrix} 1 \\ X_1 \\ X_2 \\ \vdots \\ X_p\end{bmatrix}$$ and $$\hat{\boldsymbol{\beta}} = \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \\ \vdots \\ \hat{\beta}_p \end{bmatrix}\text{,}$$ take $$\hat{Y} = \mathbf{X}^{T}\hat{\boldsymbol{\beta}} = \langle \mathbf{X}, \hat{\boldsymbol{\beta}} \rangle\text{.}$$ So $\hat{Y} \in \mathbb{R}^1$ because $\mathbf{X}^{T} \in M_{1 \times p}(\mathbb{R})$ and $\boldsymbol{\beta} \in M_{p \times 1}(\mathbb{R})$. Fine.

Now they say:

... in general $\hat{Y}$ can be a $K$-vector, in which case $\beta$ (I think this is a typo - they should probably have $\hat{\beta}$ instead) would be a $p \times K$ matrix of coefficients.

Well, here's the problem. If $\mathbf{X}^{T} \in M_{1 \times p}(\mathbb{R})$ and $\hat{\beta} \in M_{p \times K}(\mathbb{R})$, wouldn't $\mathbf{X}^{T}\hat{\beta}$ give a row vector for $\mathbf{Y}$ (dimensions $1 \times K$)?

  • 1
    $\begingroup$ Your $X$ makes no sense in this context unless its elements are interpreted as row vectors of dimension $K$. Both $\beta$ and its estimate $\hat\beta$ are column vectors of dimension $p+1$, as you show. Have you perhaps mixed up the roles of $X$ and $\beta$ in the model? $\endgroup$
    – whuber
    Commented Nov 13, 2015 at 18:20

1 Answer 1


I think you confuse yourself a bit. In page 10 the authors say:

"a set of $N$ input $p$-vectors $x_i$ , $i = 1, \dots, N$ would be represented by the $N \times p$ matrix $X$."

This means that the $p$ feature vectors/regressors they use will be represented as column vectors in the matrix $X$. For that matter and to quote from the text page 12 exactly :

"$X$ is an $N \times p$ matrix with each row an input vector".

So $X^T$ is not a row (or column) vector in any case. $X$ (and its transpose) will be (probably) of rank $p$. $\hat{Y}$ will be a matrix $N \times K$. For the case of OLS model $K$ is usually equal to unity (1) but this is not a strict prerequisite for its use (that's why usually we write $\hat{y}$).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.