$A^TA$ need not be invertible. If columns of a matrix A are linearly independent, then $A^TA$ is invertible.

Let's say $A^TA$ isn't invertible:

Gauss-Markov seems to hinge on $A^TA$ being invertible. Suppose that A is $mxn$ real matrix. If $m<n$, then the inverse of $A^TA$ does not exist.

But you can minimize the sum of squared differences, regardless (use some iterative algorithm). The proof for Gauss-Markov that I know won't work anymore. (BLUE: best linear unbiased estimator, OLS: ordinary least squares). In this situation I could see having multiple solutions. And for any of these solutions, are they the OLS? Are they BLUE? Why, if Gauss-Markov doesn't work anymore.

Gauss-Markov: https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem Convexity of Linear Regression: Convexity of linear regression

  • $\begingroup$ The last few paragraphs of this question wander off into what looks like a triviality or something without meaning: typically, $A$ has many more rows than columns, so it doesn't even make sense to refer to its "inverse." What do you really want to ask? $\endgroup$ – whuber Apr 28 '17 at 15:35
  • $\begingroup$ Thank you for the edit. Have you investigated the standard solution to least squares? You seem to be aware it can be expressed in terms of the inverse of $A^\prime A$, but since that explicitly shows there is a unique solution, it's hard to determine what you're trying to ask. $\endgroup$ – whuber Apr 28 '17 at 15:51
  • $\begingroup$ I'm trying to clarify; there maybe 2 questions here. So if $A^TA$ ISN'T invertible, can you still get the OLS solution? I think also I wasn't convinced that there was a unique solution if $A^TA$ was invertible. Does that make sense? $\endgroup$ – eternalmothra Apr 28 '17 at 16:03
  • $\begingroup$ I think the second question was too confusing, so I just removed it. I will try to restructure it and perhaps make a separate question later. $\endgroup$ – eternalmothra Apr 28 '17 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.