# Understanding linear projection in “The Elements of Statistical Learning”

In the book "The Elements of Statistical Learning" in chapter 2 ("Linear models and least squares; page no: 12"), it is written that

In the (p+1)-dimensional input-output space, (X,Y) represent a hyperplane. If the constant is included in X, then the hyperplane includes the origin and is a subspace; if not, it is an affine set cutting the Y-axis at the point (0,$\beta$).

I don't get the sentence "if constant is ... (0,$\beta$)". Please help? I think the hyperplane would cut the Y-axis at (0,$\beta$)in both the cases, is that correct?

The answer below has helped somewhat, but I am looking for more specific answer. I understand that when $1$ is included in the $X$, it won't contain origin, but then how would the $(X,Y)$ would contain origin? Should not it depend on value of $\beta$? If intercept $\beta_0$ is not $0$, $(X,Y)$ should not contain origin, in my understanding?

• How much linear algebra have you done? Do you know what vectors are? What about vector spaces, subspaces, ... ? – Adrian Dec 8 '15 at 8:06
• I have basic understanding of linear algebra, vector and vector spaces. – Abhinav Gupta Dec 8 '15 at 8:09
• en.wikipedia.org/wiki/Hyperplane has a bit on affine hyperplanes and vector hyperplanes – Adrian Dec 8 '15 at 8:52
• Thnaks! just read this article. But I still don't understand how can one say that hyperplane includes the origin if contant is included in X. If this clear then I understand why hyperplane is a subspace. – Abhinav Gupta Dec 8 '15 at 9:06
• page no: 12. I have edited the question too. – Abhinav Gupta Dec 10 '15 at 9:53

Including the constant 1 in the input vector is a common trick to include a bias (think about Y-intercept) but keeping all the terms of the expression symmetrical: you can write $\beta X$ instead of $\beta_0 + \beta X$ everywhere.

If you do this, it is then correct that the hyperplane $Y = \beta X$ includes the origin, since the origin is a vector of $0$ values and multiplying it for $\beta$ gives the value $0$.

However, your input vectors will always have the first element equal to $1$; therefore they will never contain the origin, and will be place on an smaller hyperplane, which has one less dimension.

You can visualize this by thinking of a line $Y=mx+q$ on your sheet of paper (2 dimensions). The corresponding hyperplane if you include the bias $q$ your vector becomes $X = [x, x_0=1]$ and your coefficients $\beta = [m, q]$. In 3 dimensions this is a plane passing from the origin, that intercepts the plane $x_0=1$ producing the line where your inputs can be placed.

• I still don't quite understand. The book says "If the constant is included in X, then the hyperplane includes the origin and is a subspace" but as you stated, "input vectors will always have first element = 1, so will never contain origin. So how can including constant 1 include the origin as the book says? – MinYoung Kim Aug 6 '19 at 11:23
• It also took me some time to see it but this answer helped. You have to forget about the constraint on $x_0$ to see the subspace/plan they talk about which include the origin. When you add the constrain $x_0 = 1$ you get the exact same 2d line just projected on the plan $x_0 = 1$. – grll Apr 16 '20 at 10:15
• -1 I agree with @MinYoungKim: This answer doesn't really explain why having $X_0 = 1$ means that the hyperplane includes the origin, and nor does it explain why it is an affine set cutting the $Y$-axis at the point $(0, \hat{\beta}_0)$ if the constant is not included in $X$. – The Pointer Sep 2 '20 at 20:37

Let's say we have a one dimensional problem (p=1) so a single feature (input variable) $$X_1$$ to predict a single output variable $$Y$$. Let's imagine that we already found an intercept $$\beta_0 = 5$$ and a coefficient $$\beta_1 = 2$$ for our input variable $$X_1$$.

Our linear model would look like: $$\hat{Y} = \beta_0 + \beta_1 \times X_1$$.

Hence the obvious representation would be an hyperplane (a line) in (p+1)-dimensional space in this case (2d): Another representation would be to add another variable $$X_0$$ which will lead to the following equation: $$\hat{Y} = \beta_0 \times X_0 + \beta_1 \times X_1$$.

In practice we know that $$X_0$$ will be a constant and equal to 1 but let's assume it is not fixed yet. In that case, we can now plot a 3d graph with an hyperplane as follow:

Finally since we know only $$X_0 = 1$$ is possible I highlighted with a red dashed line the only working projection out of this hyperplane which correspond exactly to the plot we had before.

• The visualisations are nice, but this answer does not explain why having $X_0 = 1$ means that the hyperplane includes the origin. – The Pointer Sep 2 '20 at 20:25
• It has been a while since I did that but you can see that the equation defining the plane in green ($\hat{Y} = \beta_0 \times X_0 + \beta_1 \times X_1$) will always include the origin – grll Sep 4 '20 at 18:58
• But that’s not an explanation. – The Pointer Sep 4 '20 at 19:02
• looking at the equation mentioned when $X_0 = 0$ and $X_1 = 0$ then $\hat{Y}$ must be 0 hence it has to include the origin – grll Sep 4 '20 at 19:05
• that was also what was confusing me and the reason why I made the visualisation. He is talking about a plane that will never exists because eventually $X_0$ will be equal to 1 but if you consider the 3 variables $\hat{Y}$, $X_0$, $X_1$ with the plane defined by the equation above you see that this plane must include the origin: (0,0,0) – grll Sep 4 '20 at 19:10

I believe that both of the answers here are incorrect, because the textbook itself is incorrect, so they're trying to justify an incorrect concept. See this answer by user "Jean-Claude Arbaut".