6
$\begingroup$

$$\large \hat y = X\,\hat\beta \tag 1$$

seems to be (?) the most commonly encountered expression of the ordinary least square projection. The actual values of the "dependent" variable differ from the estimated $\hat y$ values, so that...

$$\large y = X\,\hat\beta + \varepsilon$$

This is the expression that appears in Wikipedia. In it $X$ is the model matrix, which typically would correspond to a $m\times n$ matrix of $m$ observations or subjects, and $n$ variables. $\hat \beta$ is an $n \times 1$ vector of coefficients; and $\ y$ and $\hat y$, the observed values (or predicted values, respectively) of the "dependent" variable, forming an $m \times 1$ vector. $\varepsilon$ is the error.

However, in the book The Elements of Statistical Learning (Second Edition) by T. Hastie, R. Tibshirani and J. Friedman, on page 11, this is expressed as:

$$\large \hat Y = X^T \,\hat\beta \tag 2$$

where $X^T$ denotes vector or matrix transpose ($X$ being a column vector). Here we are modeling a single output, so $\hat Y$ is a scalar; in general $\hat Y$ can be a $K$–vector, in which case $\beta$ would be a $p \times K$ matrix of coefficients. In the $(p + 1)$-dimensional input–output space, $(X,\hat Y)$ represents a hyperplane. If the constant is included in $X$, then the hyperplane includes the origin and is a subspace; if not, it is an affine set cutting the $Y$-axis at the point $(0,\hat\beta_0)$. From now on we assume that the intercept is included in $\hat\beta$.

$X^T$ seems most intuitive with one single vector, but it is not so straightforward when it is a model matrix. Also it seems as though the authors are including the possibility of multivariate regression.

In any event, the paragraph is far from clear to me, including the hat-less beta, and would like to ask for a "connecting" explanatory answer to clarify how both expression $(1)$ and $(2)$ are equivalent (if they are).

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ Hastie et al. in this formula and in the quoted paragraph use $X$ to refer to the vector of predictors, not the model matrix. It's a column vector, hence it needs to be transposed. In contrast, in (1) the same letter $X$ refers to the model matrix where each vector of predictors (one sample) is a row. Hence the lack of transpose. Does this resolve your confusion? $\endgroup$
    – amoeba
    Apr 15, 2016 at 14:51
  • $\begingroup$ I'd like to see how they define $X$. They are saying that $Y$ is a scalar (1x1), which means $X^T \beta$ is 1x1, which means $X^T$ is 1x$n$ and $\beta$ is $n$x1, which makes no sense. $\endgroup$
    – Jessica
    Apr 15, 2016 at 15:11
  • $\begingroup$ @Jessica: why does it make no sense? $\endgroup$
    – amoeba
    Apr 15, 2016 at 15:19
  • 2
    $\begingroup$ The very notation tells you the dimensions. E.g., in the univariate regression expression "$\hat Y = X^\prime \hat\beta$" you know $\hat\beta$ is a vector, whence it must be a column vector, say of $p$ components. Then $X^\prime$ has to be a something-by-$p$ matrix, say $n\times p$, whence $\hat Y$ has to be a column vector of $n$ components. This makes $X$ itself a $p\times n$ matrix. There's nothing left to determine. Similar reasoning for multivariate regression with $r$ responses reveals $\hat Y$ must be an $n\times r$ matrix and $\hat\beta$ a $p\times r$ matrix. $\endgroup$
    – whuber ♦
    Apr 15, 2016 at 15:28
  • 1
    $\begingroup$ @Jessica Perhaps you were confused because Antoni used $n$ to refer to the number of variables. This is very unconventional, Antoni; letter $n$ almost exclusively refers to the number of samples/observations/subjects. $\endgroup$
    – amoeba
    Apr 15, 2016 at 22:13

1 Answer 1

Reset to default
4
$\begingroup$

In matrix notation $$\hat{Y} = X \hat{\beta} $$

where $\hat{Y}$ is the fitted $m \times 1$ response vector, $X$ is an $m \times n$ model matrix and $\hat{\beta}$ is the estimated $n \times 1$ coefficient. Each column of $X$ is a predictor and each row is observed predictor values for each observation.

If we to write $X$ as $(x_1^T \, x_2^T \, \ldots \, x_m^T)$ then each $x_i$ is the observed predictors for the $i$th observation. Each $x_i^T$ makes the rows of $X$. Then from the matrix notation, for each observation $i$, we have

$$\hat{y}_i = x_i^T \hat{\beta}.$$

As amoeba pointed out, Hastie et al. use the notation $X$ in place of my $x_i$ here, which is different from the $X$ notation in the first equation.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Is this at odds with @amoeba comment on the OP? $\endgroup$
    – Antoni Parellada
    Apr 15, 2016 at 15:23
  • $\begingroup$ @AntoniParellada No, it is the same as amoeba's comment clarified with equations. $\endgroup$
    – Greenparker
    Apr 15, 2016 at 15:24
  • 1
    $\begingroup$ @Antoni I agree (+1). The only thing this answer does not explicitly say is that Hastie et al. in this formula use symbol $X$ to refer to $x_i$ (or rather just $x$) in the Greenparker's notation. $\endgroup$
    – amoeba
    Apr 15, 2016 at 15:42
  • $\begingroup$ I'd like to accept your answer. Can you please reflect @amoeba comment in an edit if you think there is room for it? Ty $\endgroup$
    – Antoni Parellada
    Apr 15, 2016 at 20:54
  • 1
    $\begingroup$ @AntoniParellada I have included some more discussion of it at the end of my answer. $\endgroup$
    – Greenparker
    Apr 15, 2016 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.