3
$\begingroup$

BFGS and LBFGS algorithms are often seen used as optimization methods for non-linear machine learning problems such as with neural networks back propagation and logistic regression.

My question is why arent they implemented in everything that gradient descent is even remotely related to, LINEAR regression for example?

is it simply a matter of overkill and an unnecessary piece of machinery in this case (but it would still theoretically improve training time) or is there an actual compatibility issue where BFGS and LBFGS needs non-linearity for them to work?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
5
$\begingroup$

With linear regression, BFGS and LBFGS would be a major step backwards. That's because the solution can be directly written as

$\hat \beta = (X^T X)^{-1} X^T Y$

It's worth noting that directly using the above equation to calculate $\hat \beta$ (i.e. inverting $X^T X$ and then multiplying by $X^T Y$) is itself even a poor way to calculate $\hat \beta$.

Also, gradient descent is only recommended for linear regression in extremely special cases, so I wouldn't say gradient descent is "related" to linear regression.

Improve this answer
$\endgroup$
7
  • $\begingroup$ sorry, I think it was poor phrasing on my part. I know it would be a major step back computationally, but my major concern would be whether or not it would still work (for educational purposes and semantic understanding) the reason why know one uses is it is redundant to the point of being pointless, right? not due to some mathematical underlying incompatibility yes? $\endgroup$
    – AlanSTACK
    Commented Jan 19, 2016 at 22:27
  • $\begingroup$ @user3810748: yes, it would work. $\endgroup$
    – Cliff AB
    Commented Jan 19, 2016 at 22:31
  • $\begingroup$ also, can you elaborate on what you mean by "Also, gradient descent is only recommended for linear regression in extremely special cases, so I wouldn't say gradient descent is "related" to linear regression." do you mean that linear regression is weak compared to other better models or do you mean that gradient descent should not be used to train linear regression???(im guessing the former due to the strangeness of the latter) $\endgroup$
    – AlanSTACK
    Commented Jan 19, 2016 at 22:33
  • 1
    $\begingroup$ @user3810748: gradient descent is a generic algorithm for a local critical point of a function (hopefully the minimum!). It can be used in the specific case of linear regression, but except for very extreme cases, it's relatively very inefficient in regards to total required computation time. $\endgroup$
    – Cliff AB
    Commented Jan 19, 2016 at 22:53
  • $\begingroup$ @CliffAB are there experiments/literature regarding the generally worse performance of gradient descent versus the analytical solution? I think that the (X^T X)^-1 operation could be very costly for moderately large X and in these cases LBFGS could be maybe faster. $\endgroup$
    – Jacques Wainer
    Commented Feb 3, 2016 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.