Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Preliminaries

For simplicity's sake assume we are dealing with a 2-dimensional dataset of examples $(x_i, y_i) \in \mathbb{R}^2$ which are split into a training (objects and their label known) and test set (only objects known).

Prediction using least squares can be represented using the following matrix procedure:

  1. From our training set find the optimal regressor constant $w = Y'X(X'X)^{-1}$
  2. Apply the above to successive unlabeled examples from the test set to find their label $\hat{y} = w'x = Y'X(X'X)^{-1}x$

The Question

After you've finished part 1. of the above procedure how do you find the optimal linear least squares line?

If a linear line can be represented using $y = w'x + c$ how do you find the constant $c$?

share|improve this question

1 Answer 1

up vote 9 down vote accepted

Often in textbooks and literature the authors implicitly add a constant column of 1s to the data matrix $X$. If your original vector $x$ looks like

$$ x = \begin{bmatrix} \frac{5}{6} \\ \frac{1}{6} \\ 3 \end{bmatrix} $$

then your augmented $X$ is something like

$$ X = \begin{bmatrix} \frac{5}{6} & 1 \\ \frac{1}{6} & 1 \\ 3 & 1 \end{bmatrix} $$

Thus your $w$ vector is really a 2d vector: the first component represent the coefficient of $x$, and the second component represents the coefficient of 1 (otherwise known as the constant, $c$.)

If we denote $w = [w_1, w_2]$, then, your optimal least squares line is given by

$$ y = w_1 x + w_2$$

(Though to be consistent with existing literature, often they denote the constant coefficient $w_0$ and not $w_2$).

share|improve this answer
    
This implicit addition of a constant column of 1s is only relevant in plotting, and not the actual procedure, right? –  Helix Mar 5 '13 at 22:03
1  
Yes and no. Of course you can do the procedure without the added column, but then you are no longer modeling a linear line with a constant term, $c$. By adding the column, you are adding a constant term to the model, hence can determine $c$. Does that help? –  Cam.Davidson.Pilon Mar 5 '13 at 22:05
    
Yes it does :) thank you! –  Helix Mar 5 '13 at 22:11
1  
Both the original question and this response use nonstandard notation in respect of the great bulk of the statistical literature. The answer is correct (and +1 for it), but if someone's trying to understand regression by looking it up elsewhere and comparing it with this, the various mismatches may be confusing. In particular, in statistics, the constant is normally listed first, and the OP's $w$ is normally written as a column vector, not a row vector and would be denoted by $\hat{\beta} (= w')$, or sometimes $b$, so that $\hat{\beta} = (X'X)^{-1}X'y$. Hopefully this may clarify. –  Glen_b Mar 5 '13 at 22:56
    
@Glen_b I agree, and I used such letters to aid the OP. Your comment further reminds me to change the variables used. –  Cam.Davidson.Pilon Mar 6 '13 at 1:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.