0
$\begingroup$

In my book I have here that $\hat\beta=(X'X)^{-1}X'Y$, and that's fine and dandy, but I have a maybe dumb question regarding this.

So these $β$s are the coefficients that we must obtain from our model, but is this saying that we can obtain these coefficients simply using the formula above?

If yes, isn't this basically what we're trying to do in regression? Aren't we simply trying to find a good model and find the perfect values for the coefficients to get what we want, and we can figure that out through just this simple equation?

I guess I'm just a little confused about what we're trying to do here, if none of this makes sense let me know and it'll try to re-word it to the best of my ability

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Yes, one way to derive OLS is to show that it is the solution to the least squares problem: $\mathrm{argmin}_\beta\, \sum_{i=1}^n (y_i - x_i^T \beta)^2$. When we solve this minimization problem, we get exactly $\hat\beta = (X'X)^{-1}X'Y$. In a sense, this is trivial, since this formula just follows directly from the problem we set up. But looked at another way, that's the beauty of math: it allows us to precisely state a problem (choose a line that minimizes the sum of squared errors) and argue from this problem definition to a precise answer. Is there something else you were confused about? $\endgroup$
    – stats_model
    Commented Apr 9, 2019 at 3:28
  • $\begingroup$ @stats_model Oh wow I see that's awesome, thank you! I think that's all that I was confused about for now, if I have another question I shall be sure to post it :) $\endgroup$
    – Hello Mellow
    Commented Apr 9, 2019 at 3:45

1 Answer 1

Reset to default
0
$\begingroup$

Yes, the quoted formula can be used to calculate the regression coefficients.

However, in practice, other algorithms are used. The quoted formula is exactly the solution of the ordinary least squares (OLS) problem, as I am sure it says in you textbook.

It is important to note that OLS is a type of problem often used to find a "line of best fit" (ie, linear regression) but also encountered in computation linear algebra in other contexts. There are many algorithms used to solve the OLS problem, perhaps the simplest of which is the formula quoted. However, it is often not the best way to compute it, as it may suffer from numerical instability and performance issues. Other methods exist, such as matrix factorization approaches (eg QR decomposition and Cholesky factorization) and singular value decomposition. See these questions and their answers for more detail:

Why not use the "normal equations" to find simple least squares coefficients?

What algorithm is used in linear regression?

Least Squares Regression Step-By-Step Linear Algebra Computation

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.