# Help understanding Linear Model in ESL book

Also known as "Nate slowly deciphers ESL to conceptual understanding/plainer language", part two (see part one) Help me understand this (bullets added)

The term $\hat{β}_0$ is the intercept, also known as the bias in machine learning. Often it is convenient to include the constant variable 1 in $X$, include $\hat{β}_0$ in the vector of coefficients $\hat{β}$, and then write the linear model in vector form as an inner product $\hat{Y}= X^\top\hat{β}$ where $X^\top$denotes vector or matrix transpose (X being a column vector). Here we are modeling a single output, so $\hat{Y}$ is a scalar; in general $\hat{Y}$ can be a $K$–vector, in which case $β$ would be a $p \times K$ matrix of coefficients. In the $(p + 1)$-dimensional input–output space, $(X, \hat{Y} )$ represents a hyperplane.

1. If the constant is included in $X$, then the hyperplane includes the origin and is a subspace;
2. if not, it is an affine set cutting the $Y$-axis at the point $(0, \hat{β}_0)$.

From now on we assume that the intercept is included in $\hat{β}$.

My questions:

1. $K$ in the context of a $K$-vector and $p \times K$ matrix of coefficients -- that value is obviously different than $p$; is it different than $N$ -- the number of observations?
2. What does the notation $(X, \hat{Y} )$ mean?
3. How do they mean "hyperplane"? For example, in the the 2-D example?
4. How do they mean "subspace"? For example, in the the 2-D example?
5. How do they mean "affine set"? For example, in the the 2-D example?
6. When they "Assume the intercept is included" -- which did they choose, option 1 or option 2?
7. How do they mean "$\hat{β}_0$ is the bias" -- why is that word used? Is it related to bias vs variance?
8. Is there indeed a typo as suggested in the quoted part here ; should it be (... in which case $\mathbf{\hat{β}}$ would be a $p \times K$ matrix of coefficients...) In other words, when can $β$ take his hat off ?

1. Yes, $p$ is number of columns/variables (i.e. age, weight) $N$ is the number of rows/observations (i.e. Andy, Olly -- though this linear model operates on one-row-at-a-time) , then $K$ is yet another (orthogonal?) axis (i.e. Andy's age, and weight at age 3, age 10, age 20)?
2. It looks like a Cartesian coordinate, but it's generic (uses $X$ and $\hat{Y}$). In 2-D , (1,2) represents the point on a 2-D graph. So does (x,y) represent a set of points; i.e. a line? I have trouble reconciling a scalar $x$ with a vector $X$
3. hyperplane; They mean it cuts the (..."space"?) into two parts. In 2-D , a line cuts (.... ${\rm I\!R^2}$ ?) space on a graph into two separate portions. Indeed, that's the whole point of the "linear model" binary classification (two parts) ; could be called a "hyperplane model" for higher dimensions.
4. subspace; not sure. How can I think of "If the constant is included in $X$", in 2-D space, where X is just a scalar? Do they mean the hyperplane coincides with a plane formed by the intersection of p dimensions? In 2-D, like a line x=0 or y=0?
5. affine set; they mean it does not have an origin? Because the "intercept" has moved it away from the origin, like the "$b$" in $y=mx+b$ ? I am naive about "affine":
6. They've gone with option 1; where the constant ($1$) is included in $X$ and the intercept ($\hat{β}_0$) is included in $\hat{β}$
7. "Bias" and "weight vector" are two relevant terms here explained at the link... I am trying to understand why they would use the word "bias"; In 2-D space the "bias" the y-intercept... is it because when "x" is 0, we know we can't actually estimate "y", but a bias suggests there is some non-zero value for "y"? It is different than bias vs variance (...?)
8. Yes, it should be $\mathbf{\hat{β}}$ -- we only remove the hat when we start "viewing this as a function" (i.e. $f(X) = X^\topβ$ )

1: In case the outputs we are trying to model are real numbers then $$K = 1$$. But if we output is a GPS coordinate, then we would possibly model the output space as $$\mathbb{R}^2$$. In that case $$K = 2$$. We would have

$$(X_1, X_2, X_3) \cdot \left(\begin{array}{cc} \hat\beta^0_0 & \hat\beta^1_0 \\ \hat\beta^0_1 & \hat\beta^1_1 \\ \vdots & \vdots \\ \hat\beta^0_N & \hat\beta^1_N \\ \end{array}\right) = (\hat Y^0, \hat Y^1)$$

where the $$N \times 2$$ matrix is the $$p \times K$$ matrix that is discussed in the book.

2: Consider the simple example where $$p = 2$$. We are given $$\hat \beta$$. We have $$\hat Y = \hat \beta_0 + X_1 \hat \beta_1 + X_2 \hat \beta_2$$. Then $$(X, \hat Y)$$ can be see as $$\{ (x_1, x_2, \hat y) \} = \{ (x_1, x_2, \hat \beta_0 + x_1 \hat \beta_1 + x_2 \hat \beta_2) \} \subset \mathbb{R}^3$$

3: That's a decent guess. A vector hyperplane $$H$$ of a vector space $$V$$ of dimension $$n$$ is a linear subspace that has dimension $$n-1$$. Because it is a subspace it must include the origin (since every vector space must contain the zero vector). An affine hyperplane is just a translation of a vector hyperplane: $$a + V = \{\ a + v\ |\ v \in V\ \}$$. To be concrete, any line in $$\mathbb{R}^2$$ is an affine hyperplane (since any line is a translation of some line through the origin) but only lines through the origin are vector hyperplanes.

4,5,6:

I genuinely think they made an error in the book and said the reverse of what they were trying to say. I think they mean to say that if the intercept is included in $$\hat \beta$$ that it's an affine subspace and that if the intercept is not included in $$\hat \beta$$ that it is a linear subspace. For example, in one dimension, $$y = ax$$ is a linear subspace (missing bias $$b$$) whereas $$y = ax + b$$ is an affine subspace since it doesn't contain the origin. But I'd like to be corrected if I interpreted this incorrectly.

7: With bias, they just mean the $$\hat \beta_0$$ term, the $$y$$-intercept. You shouldn't confuse it with other uses of the word bias.

8: Not a typo. They just mean $$\beta$$ in general. They're just talking about the specific shape of $$\beta$$, not any particular $$\hat \beta$$.