2
$\begingroup$

Gradient descent is used for linear and logistic regression. It seems to me that could be used equally well for estimating the coefficients of an ARIMA model once the order parameters (p,q,d) have been determined.

But it seems that Maximum Likelihood Estimation (MLE) is the preferred method.

Why is Gradient Descent not used? What advantages does MLE have over GD and SGD ?

Improve this question
$\endgroup$
4
  • 10
    $\begingroup$ You seem to have a fundamental misunderstanding about the nature of the objects involved. MLE is a statistical method for estimating parameters; it transforms a question about unknown quantities into an optimization problem. It's a way of defining a loss function which is motivated by probability theory. Gradient descent is a numerical optimization method; it's used to solve optimization problems. You can use GD to solve MLE problems. $\endgroup$
    – Chris Haug
    Jan 3, 2018 at 19:18
  • 1
    $\begingroup$ @ChrisHaug so you are saying that MLE helps determine which error function we want to optimize? $\endgroup$
    – Skander H.
    Jan 3, 2018 at 20:06
  • 2
    $\begingroup$ Yes, it is a principled way of defining an error function for a given problem, which has certain nice statistical properties. In fact, many popular machine learning loss functions were either historically derived from MLE considerations first, or can be interpreted as being equivalent to MLE for a certain likelihood function. You have probably been using MLE without knowing it. $\endgroup$
    – Chris Haug
    Jan 3, 2018 at 20:42
  • $\begingroup$ @ChrisHaug I'd love to have your input on this question then: stats.stackexchange.com/questions/321442/… $\endgroup$
    – Skander H.
    Jan 4, 2018 at 0:19

2 Answers 2

Reset to default
9
$\begingroup$

MLE, or Maximum Likelihood Estimation, is not an algorithm, but rather a method for estimating. You can often use something like gradient descent to compute the maximum likelihood estimate, but MLE is not a computational algorithm like gradient descent.

When the number of parameters to compute is relatively low (i.e. less than hundreds), it tends to be a very bad method compared other very basic methods such as Newton's method, hence why it's not typically used.

Improve this answer
$\endgroup$
1
2
$\begingroup$

Actually packages like forecast and statsmodels in R and Python use gradient descent to estimate parameters of MLE or conditional sum of square in ARIMA model. In forecast, the defualt method is Broyden-Fletcher-Goldfarb-Shanno method, while in statsmodels, it is limited memory Broyden-Fletcher-Goldfarb-Shanno method.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.