0
$\begingroup$

Is it ever possible for least-squares linear regression (linear in both features and weights) NOT to produce a solution? That is, after we set each partial derivative to zero, can the resulting system of n linear equations in n weights be "inconsistent"? An example of "inconsistent system of linear equations" is two parallel lines (for 2 weights) or two parallel planes (for 3 weights). If the answer is yes, can you please give a simple numerical example with 2 or 3 data points?

$\endgroup$
1
  • $\begingroup$ Differentiability is not necessary--it's just a distraction. A sum of squares always has a global minimum, because it is a continuous function, bounded below by zero, with an upper bound on any compact set. Thus, if you want there to be exceptions you will have to modify the problem by introducing constraints on the solution. $\endgroup$
    – whuber
    Oct 20, 2019 at 17:44

4 Answers 4

5
$\begingroup$

A linear system $Ax=b$ can have no-solution case due to the inconsistencies introduced in its equations. But, linear regression does not aim to satisfy $Ax=b$, it merely tries to minimize $||Ax-b||_2$. This is a simple optimization problem, and you'll have your solution, either unique or infinitely many. Intuitively, one can always draw a line as best fit to a given set of points.

$\endgroup$
0
$\begingroup$

You probably mean "unique" solution not inconsistent which is a different concept. Yes, linear regression problem can have degenerated solution, i.e. multiple solutions equally good in a sense of the lowest sum of squared residuals.

A simple example is to have two identical variables in the equation, such as a temperature in Fahrenheit and Celsius. The numbers will be different in columns, but the one can be obtained from another by a trivial linear transformation. This leads to an infinite number of solutions of least squares problem.

$\endgroup$
0
$\begingroup$

No. One way of thinking of a multiple regression is as the best fitting line in a multidimensional space. There is always such a line, even if the fit is terrible. Look at all possible lines - one will be best. Unless there is exact collinearity, in which case, more than one will be best.

$\endgroup$
0
$\begingroup$

If I understand the question correctly, one situation in which OLS breaks down is when covariance matrix is singular (i.e. its determinant is 0), which indicates linear dependency between columns (i.e. variables) in your dataset.

Note that if the determinant of a singular matrix is zero, matrix inverse would fail to be calculated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.