I am having some confusion regarding least squares method. Actually, least squares method is for minimizing the square of the $L_2$ norm of $Ax-b$ as given in this video lecture. However I am confused if we are given a set of points like $(x_1,y_1),(x_2,y_2)...(x_4,y_4)$. Then our model is something like

$$y=ax+b$$ where we have to find the parameters and b. So we need to minimize the square of the $L_2$ norm of


isn't it?


Minimizing $(Ax-b)^2$ is wrong; minimizing $(y-(Ax+b))^2$ is right.

  • $\begingroup$ It's in this video lecture stanford.edu/class/ee364a/videos/video01.html. This is really popular. I think I might have misinterpreted something. I just need some clarifications $\endgroup$ – user31820 Jun 19 '12 at 16:59
  • 1
    $\begingroup$ Oh, that's just notation. They're doing plain convex optimization, not linear regression, so they're defining "b" to be (y-b) in the linear regression case. In the problem they're solving, there is no "y". Is that clear? They're just solving for x; A and b are assumed known. (edit: last statement was wrong) $\endgroup$ – Johann Hibschman Jun 19 '12 at 17:02
  • $\begingroup$ Can you give me a good working example? $\endgroup$ – user31820 Jun 19 '12 at 17:11
  • 2
    $\begingroup$ They're not solving linear regression. They're doing a math problem. There's no need for them to mention "y" in this context at all. The only problem they have is to minimize Ax-b. Linear regression just happens to be a problem you can reduce to that form. "A" becomes a matrix of [[1 x1] [1 x2] [1 x3] [1 x4]]; "x" is the vector of [b a]; "b" is y. $\endgroup$ – Johann Hibschman Jun 19 '12 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.