Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top


For simplicity's sake assume we are dealing with a 2-dimensional dataset of examples $(x_i, y_i) \in \mathbb{R}^2$ which are split into a training (objects and their label known) and test set (only objects known).

Prediction using least squares can be represented using the following matrix procedure:

  1. From our training set find the optimal regressor constant $w = Y'X(X'X)^{-1}$
  2. Apply the above to successive unlabeled examples from the test set to find their label $\hat{y} = w'x = Y'X(X'X)^{-1}x$

The Question

After you've finished part 1. of the above procedure how do you find the optimal linear least squares line?

If a linear line can be represented using $y = w'x + c$ how do you find the constant $c$?

share|improve this question
up vote 11 down vote accepted

Often in textbooks and literature the authors implicitly add a constant column of 1s to the data matrix $X$. If your original vector $x$ looks like

$$ x = \begin{bmatrix} \frac{5}{6} \\ \frac{1}{6} \\ 3 \end{bmatrix} $$

then your augmented $X$ is something like

$$ X = \begin{bmatrix} \frac{5}{6} & 1 \\ \frac{1}{6} & 1 \\ 3 & 1 \end{bmatrix} $$

Thus your $w$ vector is really a 2d vector: the first component represent the coefficient of $x$, and the second component represents the coefficient of 1 (in your example, this is denoted $c$.)

If we denote $w = [w_1, w_0]$, then, your optimal least squares line is given by

$$ y = w_1 x + w_0$$

share|improve this answer
This implicit addition of a constant column of 1s is only relevant in plotting, and not the actual procedure, right? – Helix Mar 5 '13 at 22:03
Yes and no. Of course you can do the procedure without the added column, but then you are no longer modeling a linear line with a constant term, $c$. By adding the column, you are adding a constant term to the model, hence can determine $c$. Does that help? – Cam.Davidson.Pilon Mar 5 '13 at 22:05
Yes it does :) thank you! – Helix Mar 5 '13 at 22:11
Both the original question and this response use nonstandard notation in respect of the great bulk of the statistical literature. The answer is correct (and +1 for it), but if someone's trying to understand regression by looking it up elsewhere and comparing it with this, the various mismatches may be confusing. In particular, in statistics, the constant is normally listed first, and the OP's $w$ is normally written as a column vector, not a row vector and would be denoted by $\hat{\beta} (= w')$, or sometimes $b$, so that $\hat{\beta} = (X'X)^{-1}X'y$. Hopefully this may clarify. – Glen_b Mar 5 '13 at 22:56
@Glen_b I agree, and I used such letters to aid the OP. Your comment further reminds me to change the variables used. – Cam.Davidson.Pilon Mar 6 '13 at 1:03

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.