12
$\begingroup$

In the book "The Elements of Statistical Learning" in chapter 2 ("Linear models and least squares; page no: 12"), it is written that

In the (p+1)-dimensional input-output space, (X,Y) represent a hyperplane. If the constant is included in X, then the hyperplane includes the origin and is a subspace; if not, it is an affine set cutting the Y-axis at the point (0,$\beta$).

I don't get the sentence "if constant is ... (0,$\beta$)". Please help? I think the hyperplane would cut the Y-axis at (0,$\beta$)in both the cases, is that correct?

The answer below has helped somewhat, but I am looking for more specific answer. I understand that when $1$ is included in the $X$, it won't contain origin, but then how would the $(X,Y)$ would contain origin? Should not it depend on value of $\beta$? If intercept $\beta_0$ is not $0$, $(X,Y)$ should not contain origin, in my understanding?

$\endgroup$
5
  • 2
    $\begingroup$ How much linear algebra have you done? Do you know what vectors are? What about vector spaces, subspaces, ... ? $\endgroup$
    – Adrian
    Commented Dec 8, 2015 at 8:06
  • $\begingroup$ I have basic understanding of linear algebra, vector and vector spaces. $\endgroup$ Commented Dec 8, 2015 at 8:09
  • 1
    $\begingroup$ en.wikipedia.org/wiki/Hyperplane has a bit on affine hyperplanes and vector hyperplanes $\endgroup$
    – Adrian
    Commented Dec 8, 2015 at 8:52
  • $\begingroup$ Thnaks! just read this article. But I still don't understand how can one say that hyperplane includes the origin if contant is included in X. If this clear then I understand why hyperplane is a subspace. $\endgroup$ Commented Dec 8, 2015 at 9:06
  • $\begingroup$ page no: 12. I have edited the question too. $\endgroup$ Commented Dec 10, 2015 at 9:53

3 Answers 3

4
$\begingroup$

Including the constant 1 in the input vector is a common trick to include a bias (think about Y-intercept) but keeping all the terms of the expression symmetrical: you can write $\beta X$ instead of $\beta_0 + \beta X$ everywhere.

If you do this, it is then correct that the hyperplane $Y = \beta X$ includes the origin, since the origin is a vector of $0$ values and multiplying it for $\beta$ gives the value $0$.

However, your input vectors will always have the first element equal to $1$; therefore they will never contain the origin, and will be place on an smaller hyperplane, which has one less dimension.

You can visualize this by thinking of a line $Y=mx+q$ on your sheet of paper (2 dimensions). The corresponding hyperplane if you include the bias $q$ your vector becomes $X = [x, x_0=1]$ and your coefficients $\beta = [m, q]$. In 3 dimensions this is a plane passing from the origin, that intercepts the plane $x_0=1$ producing the line where your inputs can be placed.

$\endgroup$
3
  • 1
    $\begingroup$ I still don't quite understand. The book says "If the constant is included in X, then the hyperplane includes the origin and is a subspace" but as you stated, "input vectors will always have first element = 1, so will never contain origin. So how can including constant 1 include the origin as the book says? $\endgroup$ Commented Aug 6, 2019 at 11:23
  • $\begingroup$ It also took me some time to see it but this answer helped. You have to forget about the constraint on $x_0$ to see the subspace/plan they talk about which include the origin. When you add the constrain $x_0 = 1$ you get the exact same 2d line just projected on the plan $x_0 = 1$. $\endgroup$
    – grll
    Commented Apr 16, 2020 at 10:15
  • $\begingroup$ -1 I agree with @MinYoungKim: This answer doesn't really explain why having $X_0 = 1$ means that the hyperplane includes the origin, and nor does it explain why it is an affine set cutting the $Y$-axis at the point $(0, \hat{\beta}_0)$ if the constant is not included in $X$. $\endgroup$ Commented Sep 2, 2020 at 20:37
3
$\begingroup$

To help you understand this I made a visualisation of a very simple case.

Let's say we have a one dimensional problem (p=1) so a single feature (input variable) $X_1$ to predict a single output variable $Y$. Let's imagine that we already found an intercept $\beta_0 = 5$ and a coefficient $\beta_1 = 2$ for our input variable $X_1$.

Our linear model would look like: $\hat{Y} = \beta_0 + \beta_1 \times X_1$.

Hence the obvious representation would be an hyperplane (a line) in (p+1)-dimensional space in this case (2d):

constant not included in x

Another representation would be to add another variable $X_0$ which will lead to the following equation: $\hat{Y} = \beta_0 \times X_0 + \beta_1 \times X_1$.

In practice we know that $X_0$ will be a constant and equal to 1 but let's assume it is not fixed yet. In that case, we can now plot a 3d graph with an hyperplane as follow:

constant included in x

Finally since we know only $X_0 = 1$ is possible I highlighted with a red dashed line the only working projection out of this hyperplane which correspond exactly to the plot we had before.

$\endgroup$
10
  • 1
    $\begingroup$ The visualisations are nice, but this answer does not explain why having $X_0 = 1$ means that the hyperplane includes the origin. $\endgroup$ Commented Sep 2, 2020 at 20:25
  • $\begingroup$ It has been a while since I did that but you can see that the equation defining the plane in green ($\hat{Y} = \beta_0 \times X_0 + \beta_1 \times X_1$) will always include the origin $\endgroup$
    – grll
    Commented Sep 4, 2020 at 18:58
  • $\begingroup$ But that’s not an explanation. $\endgroup$ Commented Sep 4, 2020 at 19:02
  • $\begingroup$ looking at the equation mentioned when $X_0 = 0$ and $X_1 = 0$ then $\hat{Y}$ must be 0 hence it has to include the origin $\endgroup$
    – grll
    Commented Sep 4, 2020 at 19:05
  • $\begingroup$ that was also what was confusing me and the reason why I made the visualisation. He is talking about a plane that will never exists because eventually $X_0$ will be equal to 1 but if you consider the 3 variables $\hat{Y}$, $X_0$, $X_1$ with the plane defined by the equation above you see that this plane must include the origin: (0,0,0) $\endgroup$
    – grll
    Commented Sep 4, 2020 at 19:10
3
$\begingroup$

I believe that both of the answers here are incorrect, because the textbook itself is incorrect, so they're trying to justify an incorrect concept. See this answer by the user Jean-Claude Arbaut.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.