0
$\begingroup$

I know that if the X matrix when using the normal equation is dependent then the transpose(X)*X will be non invertible and as a result there will be an infinite number of solutions to the problem.

I don't know why this is an issue. Why can't the normal equation still execute and then just return the first solution that it finds, even if there are many other equally good solutions?

$\endgroup$
3
  • $\begingroup$ It can and it does. Are you using any computational technique, algorithm, or software that actually produces multiple solutions? Could you give an example of its input and output? $\endgroup$
    – whuber
    Aug 25, 2018 at 13:34
  • $\begingroup$ I am currently studying Linear Algebra ocw.mit.edu/courses/mathematics/… which talks about how the transpose(X)*X matrix must be invertible for the normal equation to work. This is also reinforced here: stats.stackexchange.com/questions/69442/… where it is implied that if the matrix is not invertible then the normal equation won't work. $\endgroup$
    – tbrick
    Aug 25, 2018 at 18:14
  • $\begingroup$ I doubt that's what that course claims, because MIT professors generally know what they're talking about. In the other link I don't see that implication at all. Indeed, the highest-voted answer states early that "This equation has a single solution if $X^\prime X$ is invertible (non-singular). If it's not, you have more solutions." The other answer agrees. In either case, the Normal equations "work" and give the desired results. $\endgroup$
    – whuber
    Aug 25, 2018 at 18:54

1 Answer 1

1
$\begingroup$

The solutions to the normal equations will typically be used as estimators of coefficients in some prediction model, yes? So, when you have infinitely many solutions, you need some criterion to choose between them. That can be done, and one possibility is to choose a minimum-norm solution, have a look at

Just return the first solution that it finds is very imprecise, and might not work well in a prediction problem!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.