0
$\begingroup$

Gradient descent involves significant computational effort, whereas the method of least squares enables direct and accurate calculation. Does gradient descent offer any advantages over least squares estimation?

The only potential advantage of gradient descent that I can identify is that it allows us to use other loss functions. However, I'm not sure when and why we would need to use a different loss function, since by Gauss–Markov theorem the least squares estimates should be optimal.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ right now, i'm working on a problem with 800,000 observations and 200,000 parameters. Even just forming $\mathbf{X}$ would take about 500 terrabytes. In such situations it's not even remotely possible to use decomposition-based (direct) approaches, even if we were to use supercomputers to help (which is sometimes an option portal.nersc.gov/project/sparse/superlu). Thus, we must resort to gradient descent. $\endgroup$
    – John Madden
    Commented Jun 25, 2023 at 23:57
  • $\begingroup$ @mhdadk Thank you, this is exactly what I was looking for! I'm surprised I couldn't find this question myself given the similar title. It seems like my mistake was overestimating how easy it is to directly calculate the least squares method. $\endgroup$
    – Dawid
    Commented Jun 25, 2023 at 23:59

0

Reset to default