Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In an AR model the coefficients on the lags can be solved for using least squares. How is the MA part of ARMA solved for? Since the MA part is a sum of white noise terms I imagine that it is not solved using least squares.

share|improve this question
What do you expect the sum of white noise terms to usually be? – IMA Jan 24 '13 at 8:06
up vote 6 down vote accepted

The MA parameters are estimated in many different ways, including MLE. It involves solving a set of non-linear equations, which is why everyone eventually ends up resorting to numerical methods.

Here are a few links that should get you moving forward.

  1. One good place to start is Prof. Hyndman's textbook: Forecasting: Principles and Practice For this question, I'd start with Section 8.4, and then Section 8/7.

  2. Some of the theory behind estimating the MA thetas is in this lecture:

    Here's the idea: First, each error term is recursively estimated ($\epsilon_0$ is assumed to be zero.) by exploiting the fact that the $\epsilon$ 's are Normal white noise.

    Even after this, you have to revert to numerical methods to get the MLE.

    Specifically, look at the slides 52-55.

  3. Wolfram does a good job of explaining this here. They assume you will be using $Mathematica$, but the examples are relevant even if you are not using it.

    Specifically, look at the section on Innovations Algorithm, where they have an example. (One drawback is that the implementation details of the Innovations Estimation are not shared.)

    If you believe that your noise is zero-mean and Normally distributed, then be sure to also read the section on "Maximum Likelihood Method."

  4. Auto.Arima in R This Journal of Statistical Computing paper is well worth reading. A practical way to get moving on finding the best $P,d,q$ for ARIMA is implemented in forecast and the pseudo-code is in the JSS paper.

Hope some of these help you move forward.

share|improve this answer
  1. Maximum likelihood estimation is perhaps the most popular method. Even if you noise is not Normally distributed, you will still obtain the Quasi-MLE which is consistent for the parameter of interest. You can check the Hamilton's Time Series Analysis to see how the likelihood function is specified.

  2. Another method mentioned in the links provided by Ram is Yule-Walker method which is based on the method of moments estimation (GMM). The idea is to match moments such as mean, variance and autocovariances.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.