I know the ordinary least squares and gradient decent methods that are used to estimate the parameters (weights) in linear regression. But I am not that good at math and I can't understand the optimization process (the algorithm) or the parameter estimation of MA or ARMA.


1 Answer 1


You come up with the likelihood function $LL(\theta|X)$ which is a function of the set of parameters $\theta$ given the data set $X$. You find the set of parameters $\theta$ that maximizes the function for any given sample $X$. That's simple conceptually. There is no simple way to explain how you come up with the likelihood function.

Here's an example of the simplest MA(1) process, follow the link here. Eq.12 is the model:


The likelihood function is based on the assumption that $\varepsilon_t$ is from Gaussian distribution, AND that the sample is random. So, you could re-write this as $$\varepsilon_t=Y_t-(\mu+\theta\varepsilon_{t-1})$$

So, if you plug some arbitrary parameters $\hat\mu,\hat\theta$ and assume $\varepsilon_0=0$ you can estimate the errors recursively as shown on p.15: $$\hat\varepsilon_1=Y_1-\hat\mu$$ $$\hat\varepsilon_2=Y_2-(\hat\mu+\hat\theta\hat\varepsilon_1)$$

If you forget about likelihoods for a moment, you could try to minimize the sum of squares $\sum_t e_t^2(\hat\mu,\hat\theta)$ as a function of its parameters given the dataset $Y_t$. From optimization point of view it's the same problem as maximizing some other nonlinear function $LL$, but otherwise the least squares here is not necessarily a good idea.

The actual likelihood function is in Eq.14, and you can see that it's not that different from least squares in some way. It has negative of sum of squares and some other terms.

  • $\begingroup$ the problem is that the hardest part is the optimization algorithm. i mean the way it finds the set of parameters. $\endgroup$
    – floyd
    Jun 26, 2018 at 16:49
  • 1
    $\begingroup$ the optimization is hardest in technical sense, from stats point of view it's the trivial technicality $\endgroup$
    – Aksakal
    Jun 26, 2018 at 16:51
  • 1
    $\begingroup$ to be honest my motivation to ask this question is that i don't know how they get the values of $\varepsilon_{t-1}$ and $\varepsilon_{t-2}$ to use in parameter estimation. I know in gradient decent they have $x$ and $y$ and they use x and y for estimation but in MA(or part of ARMA) there is no $x$(which is the error terms). $\endgroup$
    – floyd
    Jun 26, 2018 at 17:01
  • $\begingroup$ @floyd sections 4 and 5 of the course Aksakal linked to covers the MA case. Section 6 case covers the ARMA case. $\endgroup$
    – Skander H.
    Jun 26, 2018 at 20:18
  • $\begingroup$ @Aksakal can you provide links to other chapters in that course? I tried backtracking the http link but got 404'd. $\endgroup$
    – Skander H.
    Jun 26, 2018 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.