8
$\begingroup$

There are a number of websites describing gradient descent to find the parameters for simple linear regression (here is one of them). Google also describes it in their new (to the public) ML course.

However on Wikipedia, the following formulae to calculate the parameters are supplied: $$ {\displaystyle {\begin{aligned}{\hat {\alpha }}&={\bar {y}}-{\hat {\beta }}\,{\bar {x}},\\{\hat {\beta }}&={\frac {\sum _{i=1}^{n}(x_{i}-{\bar {x}})(y_{i}-{\bar {y}})}{\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}}}\end{aligned}}} $$

Also, the scikit-learn LinearRegression function, does not have an n_iter_ (number of iterations) attribute as it does for many other learning functions, which I suppose suggests gradient descent isn't being used?

Questions:

  1. Are the websites describing gradient descent for simple linear regression only doing so to teach the concept of it on the most basic ML model? Is the formula on Wikipedia what most stats software would use to calculate the parameters (at least scikit-learn does not seem to be using gradient descent)?
  2. What is typically used for multiple linear regression?
  3. For what types of statistical learning models is gradient descent typically used to find the parameters over other methods? I.e. is there some rule of thumb?
$\endgroup$
8
$\begingroup$
  1. Linear regression is commonly used as a way to introduce the concept of gradient descent.

  2. QR factorization is the most common strategy. SVD and Cholesky factorization are other options. See Do we need gradient descent to find the coefficients of a linear regression model

In particular, note that the equations that you have written can evince poor numerical conditioning and/or be expensive to compute. QR factorization is less susceptible to conditioning issues (but not immune) and is not too expensive.

  1. Neural networks are the most prominent example of applied use of gradient descent, but it is far from the only example. Another example of a problem that requires iterative updates is logistic regression, which does not allow for direct solutions, so typically Newton-Raphson is used. (But GD or its variants might also be used.)
$\endgroup$
  • $\begingroup$ In the link you supplied, does #3: the "Normal equations", refer to the equations in my question here? If not, what is the technical term for these equations? $\endgroup$ – Oliver Angelil Apr 26 '18 at 21:10
  • 1
    $\begingroup$ @OliverAngelil The "normal equations" are indeed the jargon term for the linear system of equations that are the first order conditions for the ordinary least squares optimization problem. $\endgroup$ – Matthew Gunn Apr 26 '18 at 21:14
  • $\begingroup$ So are the "normal equations" used in statistical software when there's only 1 predictor variable? For n = 100, I get identical (to 6 decimal places) b0 and b1 coefficients when I use the normal equations vs the LinearRegression function in scikit-learn. Although I'm confused: #3 in the link states that the "normal equations" are a "TERRIBLE" idea?? $\endgroup$ – Oliver Angelil Apr 26 '18 at 23:45
  • 1
    $\begingroup$ 6 decimal places is more than enough for me! $\endgroup$ – Oliver Angelil Apr 27 '18 at 0:59
  • 1
    $\begingroup$ @anu Solving logistic regression in a non-iterative way requires minimizing a non-linear system of equations; in general, this is hard! This situation is analogous to the Abel-Ruffini theorem (no algebraic solution to roots of a 5th degree polynomial): we simply don't have direct computation methods to solve the system exactly. IIRC, this is discussed in Elements of Statistical Learning's chapter about logistic regression. There's probably a thread somewhere on stats.SE about it as well, but I'm having trouble finding a good one. $\endgroup$ – Reinstate Monica Dec 28 '18 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.