In the book "The Elements of Statistical Learning" in chapter 2 ("Linear models and least squares; page no: 12"), it is written that

In the (p+1)-dimensional input-output space, (X,Y) represent a hyperplane. If the constant is included in X, then the hyperplane includes the origin and is a subspace; if not, it is an affine set cutting the Y-axis at the point (0,$\beta$).

I don't get the sentence "if constant is ... (0,$\beta$)". Please help? I think the hyperplane would cut the Y-axis at (0,$\beta$)in both the cases, is that correct?

The answer below has helped somewhat, but I am looking for more specific answer. I understand that when $1$ is included in the $X$, it won't contain origin, but then how would the $(X,Y)$ would contain origin? Should not it depend on value of $\beta$? If intercept $\beta_0$ is not $0$, $(X,Y)$ should not contain origin, in my understanding?

  • 2
    $\begingroup$ How much linear algebra have you done? Do you know what vectors are? What about vector spaces, subspaces, ... ? $\endgroup$ – Adrian Dec 8 '15 at 8:06
  • $\begingroup$ I have basic understanding of linear algebra, vector and vector spaces. $\endgroup$ – Abhinav Gupta Dec 8 '15 at 8:09
  • 1
    $\begingroup$ en.wikipedia.org/wiki/Hyperplane has a bit on affine hyperplanes and vector hyperplanes $\endgroup$ – Adrian Dec 8 '15 at 8:52
  • $\begingroup$ Thnaks! just read this article. But I still don't understand how can one say that hyperplane includes the origin if contant is included in X. If this clear then I understand why hyperplane is a subspace. $\endgroup$ – Abhinav Gupta Dec 8 '15 at 9:06
  • $\begingroup$ page no: 12. I have edited the question too. $\endgroup$ – Abhinav Gupta Dec 10 '15 at 9:53

Including the constant 1 in the input vector is a common trick to include a bias (think about Y-intercept) but keeping all the terms of the expression symmetrical: you can write $\beta X$ instead of $\beta_0 + \beta X$ everywhere.

If you do this, it is then correct that the hyperplane $Y = \beta X$ includes the origin, since the origin is a vector of $0$ values and multiplying it for $\beta$ gives the value $0$.

However, your input vectors will always have the first element equal to $1$; therefore they will never contain the origin, and will be place on an smaller hyperplane, which has one less dimension.

You can visualize this by thinking of a line $Y=mx+q$ on your sheet of paper (2 dimensions). The corresponding hyperplane if you include the bias $q$ your vector becomes $X = [x, x_0=1]$ and your coefficients $\beta = [m, q]$. In 3 dimensions this is a plane passing from the origin, that intercepts the plane $x_0=1$ producing the line where your inputs can be placed.

  • 1
    $\begingroup$ I still don't quite understand. The book says "If the constant is included in X, then the hyperplane includes the origin and is a subspace" but as you stated, "input vectors will always have first element = 1, so will never contain origin. So how can including constant 1 include the origin as the book says? $\endgroup$ – MinYoung Kim Aug 6 '19 at 11:23
  • $\begingroup$ It also took me some time to see it but this answer helped. You have to forget about the constraint on $x_0$ to see the subspace/plan they talk about which include the origin. When you add the constrain $x_0 = 1$ you get the exact same 2d line just projected on the plan $x_0 = 1$. $\endgroup$ – grll Apr 16 '20 at 10:15
  • $\begingroup$ -1 I agree with @MinYoungKim: This answer doesn't really explain why having $X_0 = 1$ means that the hyperplane includes the origin, and nor does it explain why it is an affine set cutting the $Y$-axis at the point $(0, \hat{\beta}_0)$ if the constant is not included in $X$. $\endgroup$ – The Pointer Sep 2 '20 at 20:37

To help you understand this I made a visualisation of a very simple case.

Let's say we have a one dimensional problem (p=1) so a single feature (input variable) $X_1$ to predict a single output variable $Y$. Let's imagine that we already found an intercept $\beta_0 = 5$ and a coefficient $\beta_1 = 2$ for our input variable $X_1$.

Our linear model would look like: $\hat{Y} = \beta_0 + \beta_1 \times X_1$.

Hence the obvious representation would be an hyperplane (a line) in (p+1)-dimensional space in this case (2d):

constant not included in x

Another representation would be to add another variable $X_0$ which will lead to the following equation: $\hat{Y} = \beta_0 \times X_0 + \beta_1 \times X_1$.

In practice we know that $X_0$ will be a constant and equal to 1 but let's assume it is not fixed yet. In that case, we can now plot a 3d graph with an hyperplane as follow:

constant included in x

Finally since we know only $X_0 = 1$ is possible I highlighted with a red dashed line the only working projection out of this hyperplane which correspond exactly to the plot we had before.

  • $\begingroup$ The visualisations are nice, but this answer does not explain why having $X_0 = 1$ means that the hyperplane includes the origin. $\endgroup$ – The Pointer Sep 2 '20 at 20:25
  • $\begingroup$ It has been a while since I did that but you can see that the equation defining the plane in green ($\hat{Y} = \beta_0 \times X_0 + \beta_1 \times X_1$) will always include the origin $\endgroup$ – grll Sep 4 '20 at 18:58
  • $\begingroup$ But that’s not an explanation. $\endgroup$ – The Pointer Sep 4 '20 at 19:02
  • $\begingroup$ looking at the equation mentioned when $X_0 = 0$ and $X_1 = 0$ then $\hat{Y}$ must be 0 hence it has to include the origin $\endgroup$ – grll Sep 4 '20 at 19:05
  • $\begingroup$ that was also what was confusing me and the reason why I made the visualisation. He is talking about a plane that will never exists because eventually $X_0$ will be equal to 1 but if you consider the 3 variables $\hat{Y}$, $X_0$, $X_1$ with the plane defined by the equation above you see that this plane must include the origin: (0,0,0) $\endgroup$ – grll Sep 4 '20 at 19:10

I believe that both of the answers here are incorrect, because the textbook itself is incorrect, so they're trying to justify an incorrect concept. See this answer by user "Jean-Claude Arbaut".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.