Understanding linear projection in "The Elements of Statistical Learning"

Question

In the book "The Elements of Statistical Learning" in chapter 2 ("Linear models and least squares; page no: 12"), it is written that

In the (p+1)-dimensional input-output space, (X,Y) represent a hyperplane. If the constant is included in X, then the hyperplane includes the origin and is a subspace; if not, it is an affine set cutting the Y-axis at the point (0,$\beta$).

I don't get the sentence "if constant is ... (0,$\beta$)". Please help? I think the hyperplane would cut the Y-axis at (0,$\beta$)in both the cases, is that correct?

The answer below has helped somewhat, but I am looking for more specific answer. I understand that when $1$ is included in the $X$, it won't contain origin, but then how would the $(X,Y)$ would contain origin? Should not it depend on value of $\beta$? If intercept $\beta_0$ is not $0$, $(X,Y)$ should not contain origin, in my understanding?

Answer 1

Including the constant 1 in the input vector is a common trick to include a bias (think about Y-intercept) but keeping all the terms of the expression symmetrical: you can write $\beta X$ instead of $\beta_0 + \beta X$ everywhere.

If you do this, it is then correct that the hyperplane $Y = \beta X$ includes the origin, since the origin is a vector of $0$ values and multiplying it for $\beta$ gives the value $0$.

However, your input vectors will always have the first element equal to $1$; therefore they will never contain the origin, and will be place on an smaller hyperplane, which has one less dimension.

You can visualize this by thinking of a line $Y=mx+q$ on your sheet of paper (2 dimensions). The corresponding hyperplane if you include the bias $q$ your vector becomes $X = [x, x_0=1]$ and your coefficients $\beta = [m, q]$. In 3 dimensions this is a plane passing from the origin, that intercepts the plane $x_0=1$ producing the line where your inputs can be placed.

Answer 2

To help you understand this I made a visualisation of a very simple case.

Let's say we have a one dimensional problem (p=1) so a single feature (input variable) $X_1$ to predict a single output variable $Y$. Let's imagine that we already found an intercept $\beta_0 = 5$ and a coefficient $\beta_1 = 2$ for our input variable $X_1$.

Our linear model would look like: $\hat{Y} = \beta_0 + \beta_1 \times X_1$.

Hence the obvious representation would be an hyperplane (a line) in (p+1)-dimensional space in this case (2d):

Another representation would be to add another variable $X_0$ which will lead to the following equation: $\hat{Y} = \beta_0 \times X_0 + \beta_1 \times X_1$.

In practice we know that $X_0$ will be a constant and equal to 1 but let's assume it is not fixed yet. In that case, we can now plot a 3d graph with an hyperplane as follow:

Finally since we know only $X_0 = 1$ is possible I highlighted with a red dashed line the only working projection out of this hyperplane which correspond exactly to the plot we had before.

Answer 3

I believe that both of the answers here are incorrect, because the textbook itself is incorrect, so they're trying to justify an incorrect concept. See this answer by the user Jean-Claude Arbaut.