# What happens to the constant in Least Squares

Preliminaries

For simplicity's sake assume we are dealing with a 2-dimensional dataset of examples $(x_i, y_i) \in \mathbb{R}^2$ which are split into a training (objects and their label known) and test set (only objects known).

Prediction using least squares can be represented using the following matrix procedure:

1. From our training set find the optimal regressor constant $w = Y'X(X'X)^{-1}$
2. Apply the above to successive unlabeled examples from the test set to find their label $\hat{y} = w'x = Y'X(X'X)^{-1}x$

The Question

After you've finished part 1. of the above procedure how do you find the optimal linear least squares line?

If a linear line can be represented using $y = w'x + c$ how do you find the constant $c$?

-

Often in textbooks and literature the authors implicitly add a constant column of 1s to the data matrix $X$. If your original vector $x$ looks like

$$x = \begin{bmatrix} \frac{5}{6} \\ \frac{1}{6} \\ 3 \end{bmatrix}$$

then your augmented $X$ is something like

$$X = \begin{bmatrix} \frac{5}{6} & 1 \\ \frac{1}{6} & 1 \\ 3 & 1 \end{bmatrix}$$

Thus your $w$ vector is really a 2d vector: the first component represent the coefficient of $x$, and the second component represents the coefficient of 1 (in your example, this is denoted $c$.)

If we denote $w = [w_1, w_0]$, then, your optimal least squares line is given by

$$y = w_1 x + w_0$$

-
This implicit addition of a constant column of 1s is only relevant in plotting, and not the actual procedure, right? – Helix Mar 5 '13 at 22:03
Yes and no. Of course you can do the procedure without the added column, but then you are no longer modeling a linear line with a constant term, $c$. By adding the column, you are adding a constant term to the model, hence can determine $c$. Does that help? – Cam.Davidson.Pilon Mar 5 '13 at 22:05
Yes it does :) thank you! – Helix Mar 5 '13 at 22:11
Both the original question and this response use nonstandard notation in respect of the great bulk of the statistical literature. The answer is correct (and +1 for it), but if someone's trying to understand regression by looking it up elsewhere and comparing it with this, the various mismatches may be confusing. In particular, in statistics, the constant is normally listed first, and the OP's $w$ is normally written as a column vector, not a row vector and would be denoted by $\hat{\beta} (= w')$, or sometimes $b$, so that $\hat{\beta} = (X'X)^{-1}X'y$. Hopefully this may clarify. – Glen_b Mar 5 '13 at 22:56
@Glen_b I agree, and I used such letters to aid the OP. Your comment further reminds me to change the variables used. – Cam.Davidson.Pilon Mar 6 '13 at 1:03