1
$\begingroup$

BFGS and LBFGS algorithms are often seen used as optimization methods for non-linear machine learning problems such as with neural networks back propagation and logistic regression.

My question is why arent they implemented in everything that gradient descent is even remotely related to, LINEAR regression for example?

is it simply a matter of overkill and an unnecessary piece of machinery in this case (but it would still theoretically improve training time) or is there an actual compatibility issue where BFGS and LBFGS needs non-linearity for them to work?

$\endgroup$
4
$\begingroup$

With linear regression, BFGS and LBFGS would be a major step backwards. That's because the solution can be directly written as

$\hat \beta = (X^T X)^{-1} X^T Y$

It's worth noting that directly using the above equation to calculate $\hat \beta$ (i.e. inverting $X^T X$ and then multiplying by $X^T Y$) is itself even a poor way to calculate $\hat \beta$.

Also, gradient descent is only recommended for linear regression in extremely special cases, so I wouldn't say gradient descent is "related" to linear regression.

$\endgroup$
  • $\begingroup$ sorry, I think it was poor phrasing on my part. I know it would be a major step back computationally, but my major concern would be whether or not it would still work (for educational purposes and semantic understanding) the reason why know one uses is it is redundant to the point of being pointless, right? not due to some mathematical underlying incompatibility yes? $\endgroup$ – AlanSTACK Jan 19 '16 at 22:27
  • $\begingroup$ @user3810748: yes, it would work. $\endgroup$ – Cliff AB Jan 19 '16 at 22:31
  • $\begingroup$ also, can you elaborate on what you mean by "Also, gradient descent is only recommended for linear regression in extremely special cases, so I wouldn't say gradient descent is "related" to linear regression." do you mean that linear regression is weak compared to other better models or do you mean that gradient descent should not be used to train linear regression???(im guessing the former due to the strangeness of the latter) $\endgroup$ – AlanSTACK Jan 19 '16 at 22:33
  • 1
    $\begingroup$ @user3810748: gradient descent is a generic algorithm for a local critical point of a function (hopefully the minimum!). It can be used in the specific case of linear regression, but except for very extreme cases, it's relatively very inefficient in regards to total required computation time. $\endgroup$ – Cliff AB Jan 19 '16 at 22:53
  • $\begingroup$ @CliffAB are there experiments/literature regarding the generally worse performance of gradient descent versus the analytical solution? I think that the (X^T X)^-1 operation could be very costly for moderately large X and in these cases LBFGS could be maybe faster. $\endgroup$ – Jacques Wainer Feb 3 '16 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.