2
$\begingroup$

Is it possible for more than one "linear regression" line to fit a given set of points? (i.e.... "least squares" is minimised equally in both cases)

I'm assuming a simple one-variable regression in which we have a single "dependent" variable for a given x-value. It may be that regression is forced "through the origin" for the purpose of the Q (if that matters), so, I know it wouldn't be a great fit.

Or is there something intrinsic about a regression line that means it is unique? I am interested in a 'proof' (could be just an intuitive proof) if it exists, that a regression line must be unique.

|cite|improve this question
$\endgroup$
4
$\begingroup$

No. Under usual conditions you have a unique solution to OLS equation.

You can always come up with degenerate cases, e.g. $$y=\beta_0+\beta_1 x_1+\beta_2x_2+e$$

If $x_1=x_2$, then any $\beta_1'=\beta_1-c,\beta_2'=\beta_2+c$ will work as well.

The reason is simple. Consider OLS equation: $y=X\beta+e$, you minimize squared residuals $e=y-X\beta$ (vector notation): $$e^2=y'y+\beta'X'X\beta-2y'X\beta$$ This is a quadratic (on unknown $\beta$) equation, convex, you can apply linear algebra to show that it has a unique solution. Intuitively it's convex, so there's a bottom, just one point. Again, this is under certain reasonable conditions.

Obviously, here I'm skipping the discussion of errors vs. residuals.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ What happens if you have more features than observations? $\endgroup$ – Sycorax says Reinstate Monica Mar 5 '16 at 2:11
  • $\begingroup$ @user777, you may end up with infinite number of solutions. Imagine in the model in my example you have only one observation: $y=1,x_1=1,x_2=1$, so you get $1=\beta_0+\beta_1+\beta_2$, and you can plug infinite number of combinations to satisfy it $\endgroup$ – Aksakal Mar 5 '16 at 2:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.