I have a LOT of time series observations and I would like to estimate a simple AR(1) model $$ y_t =c+ \phi y_{t-1}+ \varepsilon_t \qquad \varepsilon_t \sim \text{N}(0, \sigma^{2}) $$ with parameters $\theta =\left\{ c, \phi,\sigma^{2} \right\}$. Because of the amount of observations I would like to use stochastic gradient descent (or "deterministic" Robbins and Monro algorithm) to estimate the model $$ \theta^{i}_{t+1} = \theta^{i}_t + \gamma_t s^{i}_t , \quad \sum \gamma_t =\infty \quad \text{and}\quad \sum \gamma^{2}_t <\infty $$ where $\theta^{i}$ is an element of $\theta$ and $s^{i}_t$ is the derivative of log likelihood contribution at time t with respect to $\theta^{i}$.

I have the following questions:

  • First of all is this supposed to work? What alternative online estimation procedures are out there? I have the feeling that this is really not robust and needs a lot of case by case tuning.

  • How to set $\gamma_t$? I know the standard choice of $\gamma_t$ is $t^{-2/3}$ which satisfy the requirements. My experience so far is that I should set $\gamma_t$ differently for different parameters as the data is more informative about the constant $c$ than about $\sigma^2$ and the gradients have totally different magnitude. If i use the same gamma some of the parameters would diverge (too large step size) while others would converge slowly. Is there a "stochastic Newton-Raphson" type algorithm which would scale the gradients properly? I guess trying different combinations would work but there are too many combinations if I would set different gammas for different parameters.

Any comment is appreciated!

Thanks in advance!

  • $\begingroup$ Why are you equating SGD with Robbins-Monro? They're not the same. Robbins-Monro is in fact a type of stochastic Newton-Raphson method. $\endgroup$
    – Digio
    Nov 8, 2018 at 11:36

1 Answer 1


One assumption of stochastic gradient descent is that you should have independent identical distributed gradients, e.g. $s^i_t$ is independent over $t$, so that the law of large numbers ensures the stochastic gradient is a good approximation of the real gradient.

For $AR(1)$ and for most time series model, it is not.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.