# Transpose or lack of transpose in the $\hat y=X\hat \beta$ regression equation

$$\large \hat y = X\,\hat\beta \tag 1$$

seems to be (?) the most commonly encountered expression of the ordinary least square projection. The actual values of the "dependent" variable differ from the estimated $\hat y$ values, so that...

$$\large y = X\,\hat\beta + \varepsilon$$

This is the expression that appears in Wikipedia. In it $X$ is the model matrix, which typically would correspond to a $m\times n$ matrix of $m$ observations or subjects, and $n$ variables. $\hat \beta$ is an $n \times 1$ vector of coefficients; and $\ y$ and $\hat y$, the observed values (or predicted values, respectively) of the "dependent" variable, forming an $m \times 1$ vector. $\varepsilon$ is the error.

However, in the book The Elements of Statistical Learning (Second Edition) by T. Hastie, R. Tibshirani and J. Friedman, on page 11, this is expressed as:

$$\large \hat Y = X^T \,\hat\beta \tag 2$$

where $X^T$ denotes vector or matrix transpose ($X$ being a column vector). Here we are modeling a single output, so $\hat Y$ is a scalar; in general $\hat Y$ can be a $K$–vector, in which case $\beta$ would be a $p \times K$ matrix of coefficients. In the $(p + 1)$-dimensional input–output space, $(X,\hat Y)$ represents a hyperplane. If the constant is included in $X$, then the hyperplane includes the origin and is a subspace; if not, it is an affine set cutting the $Y$-axis at the point $(0,\hat\beta_0)$. From now on we assume that the intercept is included in $\hat\beta$.

$X^T$ seems most intuitive with one single vector, but it is not so straightforward when it is a model matrix. Also it seems as though the authors are including the possibility of multivariate regression.

In any event, the paragraph is far from clear to me, including the hat-less beta, and would like to ask for a "connecting" explanatory answer to clarify how both expression $(1)$ and $(2)$ are equivalent (if they are).

• Hastie et al. in this formula and in the quoted paragraph use $X$ to refer to the vector of predictors, not the model matrix. It's a column vector, hence it needs to be transposed. In contrast, in (1) the same letter $X$ refers to the model matrix where each vector of predictors (one sample) is a row. Hence the lack of transpose. Does this resolve your confusion? – amoeba says Reinstate Monica Apr 15 '16 at 14:51
• I'd like to see how they define $X$. They are saying that $Y$ is a scalar (1x1), which means $X^T \beta$ is 1x1, which means $X^T$ is 1x$n$ and $\beta$ is $n$x1, which makes no sense. – Jessica Apr 15 '16 at 15:11
• @Jessica: why does it make no sense? – amoeba says Reinstate Monica Apr 15 '16 at 15:19
• The very notation tells you the dimensions. E.g., in the univariate regression expression "$\hat Y = X^\prime \hat\beta$" you know $\hat\beta$ is a vector, whence it must be a column vector, say of $p$ components. Then $X^\prime$ has to be a something-by-$p$ matrix, say $n\times p$, whence $\hat Y$ has to be a column vector of $n$ components. This makes $X$ itself a $p\times n$ matrix. There's nothing left to determine. Similar reasoning for multivariate regression with $r$ responses reveals $\hat Y$ must be an $n\times r$ matrix and $\hat\beta$ a $p\times r$ matrix. – whuber Apr 15 '16 at 15:28
• @Jessica Perhaps you were confused because Antoni used $n$ to refer to the number of variables. This is very unconventional, Antoni; letter $n$ almost exclusively refers to the number of samples/observations/subjects. – amoeba says Reinstate Monica Apr 15 '16 at 22:13

In matrix notation $$\hat{Y} = X \hat{\beta}$$

where $\hat{Y}$ is the fitted $m \times 1$ response vector, $X$ is an $m \times n$ model matrix and $\hat{\beta}$ is the estimated $n \times 1$ coefficient. Each column of $X$ is a predictor and each row is observed predictor values for each observation.

If we to write $X$ as $(x_1^T \, x_2^T \, \ldots \, x_m^T)$ then each $x_i$ is the observed predictors for the $i$th observation. Each $x_i^T$ makes the rows of $X$. Then from the matrix notation, for each observation $i$, we have

$$\hat{y}_i = x_i^T \hat{\beta}.$$

As amoeba pointed out, Hastie et al. use the notation $X$ in place of my $x_i$ here, which is different from the $X$ notation in the first equation.

• Is this at odds with @amoeba comment on the OP? – Antoni Parellada Apr 15 '16 at 15:23
• @AntoniParellada No, it is the same as amoeba's comment clarified with equations. – Greenparker Apr 15 '16 at 15:24
• @Antoni I agree (+1). The only thing this answer does not explicitly say is that Hastie et al. in this formula use symbol $X$ to refer to $x_i$ (or rather just $x$) in the Greenparker's notation. – amoeba says Reinstate Monica Apr 15 '16 at 15:42
• I'd like to accept your answer. Can you please reflect @amoeba comment in an edit if you think there is room for it? Ty – Antoni Parellada Apr 15 '16 at 20:54
• @AntoniParellada I have included some more discussion of it at the end of my answer. – Greenparker Apr 15 '16 at 21:00