I read the following from a textbook, The Elements of Statistical Learning on page 54.

  1. Initiate $z_0 = x_0 = 1$

  2. For $j = 1, \dots, p$:

    Regress $x_j$ on $z_0, \dots, z_{j-1}$ to produce coefficients $\hat{\gamma}_{kj} = \langle z_l, xj\rangle / \langle z_l, z_l \rangle$ and residual vector $z_j = x_j - \sum\limits_{k=0}^{j-1} \hat{\gamma}_{kj}z_{k}$

  3. Regress $y$ on the residual $z_p$ to give the estimate $\hat{\beta}_p$.

The result of this algorithm is

$\hat{\beta}_p = \langle z_p, y\rangle / \langle z_p, z_p \rangle$ (3.28)

The book then goes on to assure the following:

Re-arranging the residual in step 2, we can see that each of the xj is a linear combination of the $z_k$, $k \leq j$. Since the $z_j$ are all orthogonal, they form a basis for the column space of $X$, and hence the least squares projection onto this subspace is $\hat{y}$. Since $z_p$ alone involves $x_p$ (with coefficient 1), we see that the coefficient (3.28) is indeed the multiple regression coefficient of $y$ on $x_p$.

I do no believe the last sentences makes sense

I think the coefficient of $y$ regressed on $x$ should be something like

$\sum\limits_{i=1}^p \eta\beta_i$.

for some constants $\eta$

$x_j = z_j + \sum\limits_{k=0}^{j-1} \hat{\gamma}_{kj}z_{k}$

Hence $z = \Gamma x$, where $\Gamma$ is a upper triangular matrix with diagonal 1.

Hence $\hat{\beta} z = \hat{\beta} \Gamma^{-1} x$ where $\Gamma^{-1}$ is a lower triangular matrix.

Hence the regression of $y$ on $x_i$

$\sum\limits_{i=p}^{n} \eta\hat{\beta}_i$.

where $\eta$ is obtained by inverting $\Gamma$ and the fact the inverse of an upper triangular matrix is a lower triangular one. Hence I disagree with the assertion in the book.


1 Answer 1


Actually, I figured this out now. The assertion is correct because it talks about the LAST coefficent, not the coefficient in general.

However, we can permute each coefficient to make it the last one, hence it holds for every coefficient that the coefficient of multilinear regression is the same as applying Gram-Schimdt to that variable with respect to all others and then regressing the $y$ as against the result.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.