# Moving-average model error terms

This is a basic question on Box-Jenkins MA models. As I understand, an MA model is basically a linear regression of time-series values $Y$ against previous error terms $e_t,..., e_{t-n}$. That is, the observation $Y$ is first regressed against its previous values $Y_{t-1}, ..., Y_{t-n}$ and then one or more $Y - \hat{Y}$ values are used as the error terms for the MA model.

But how are the error terms calculated in an ARIMA(0, 0, 2) model? If the MA model is used without an autoregressive part and thus no estimated value, how can I possibly have an error term?

• No, I think you are confusing the definition of an MA(n) model, where the regression is only in terms of the $e_{t-i}$'s, with its estimation, where the $e_{t-i}$'s are estimated from the data. Apr 7, 2012 at 17:25
• The main problem in your question is that you say that MA model is basically a linear regression. This is simply not true, since we do not observe error terms. Apr 7, 2012 at 20:18
• I think the error term is actually $Y_t - \hat{Y_t}$, where $\hat{Y}$ is $E(Y|Y_{t,...,t-n})$ or simply $Y_t - Y_{t-1}$. That is why an MA model parameter estimate is derived from a recurring pattern in the $Y$ partial autocorrelation function, that is the behavior of the residuals. The AR parameter estimation instead, is based on a recurring pattern of the acf(Y). Apr 9, 2012 at 15:23

MA Model Estimation:

Let us assume a series with 100 time points, and say this is characterized by MA(1) model with no intercept. Then the model is given by

$$y_t=\varepsilon_t-\theta\varepsilon_{t-1},\quad t=1,2,\cdots,100\quad (1)$$

The error term here is not observed. So to obtain this, Box et al. Time Series Analysis: Forecasting and Control (3rd Edition), page 228, suggest that the error term is computed recursively by,

$$\varepsilon_t=y_t+\theta\varepsilon_{t-1}$$

So the error term for $t=1$ is, $$\varepsilon_{1}=y_{1}+\theta\varepsilon_{0}$$ Now we cannot compute this without knowing the value of $\theta$. So to obtain this, we need to compute the Initial or Preliminary estimate of the model, refer to Box et al. of the said book, Section 6.3.2 page 202 state that,

It has been shown that the first $q$ autocorrelations of MA($q$) process are nonzero and can be written in terms of the parameters of the model as $$\rho_k=\displaystyle\frac{-\theta_{k}+\theta_1\theta_{k+1}+\theta_2\theta_{k+2}+\cdots+\theta_{q-k}\theta_q}{1+\theta_1^2+\theta_2^2+\cdots+\theta_q^2}\quad k=1,2,\cdots, q$$ The expression above for$\rho_1,\rho_2\cdots,\rho_q$ in terms $\theta_1,\theta_2,\cdots,\theta_q$, supplies $q$ equations in $q$ unknowns. Preliminary estimates of the $\theta$s can be obtained by substituting estimates $r_k$ for $\rho_k$ in above equation

Note that $r_k$ is the estimated autocorrelation. There are more discussion in Section 6.3 - Initial Estimates for the Parameters, please read on that. Now, assuming we obtain the initial estimate $\theta=0.5$. Then, $$\varepsilon_{1}=y_{1}+0.5\varepsilon_{0}$$ Now, another problem is we don't have value for $\varepsilon_0$ because $t$ starts at 1, and so we cannot compute $\varepsilon_1$. Luckily, there are two methods two obtain this,

1. Conditional Likelihood
2. Unconditional Likelihood

According to Box et al. Section 7.1.3 page 227, the values of $\varepsilon_0$ can be substituted to zero as an approximation if $n$ is moderate or large, this method is Conditional Likelihood. Otherwise, Unconditional Likelihood is used, wherein the value of $\varepsilon_0$ is obtain by back-forecasting, Box et al. recommend this method. Read more about back-forecasting at Section 7.1.4 page 231.

After obtaining the initial estimates and value of $\varepsilon_0$, then finally we can proceed with the recursive calculation of the error term. Then the final stage is to estimate the parameter of the model $(1)$, remember this is not the preliminary estimate anymore.

In estimating the parameter $\theta$, I use Nonlinear Estimation procedure, particularly the Levenberg-Marquardt algorithm, since MA models are nonlinear on its parameter.

Overall, I would highly recommend you to read Box et al. Time Series Analysis: Forecasting and Control (3rd Edition).

• Can you explain what is $r_k$? Jan 8, 2018 at 3:59
• can you explain me with a easy example? lets say my time series is 4,9,13,10 & i want to predict fifth element using MA(1) process. can you use pen and paper and tell me how to calculate error term? y(t) = constant + theta1 * error term (t-1) Apr 15, 2020 at 9:16

A Gaussian MA(q) model is defined (not only by Box and Jenkins!) as $$Y_t = -\sum_{i=1}^q \vartheta_i e_{t-i} + \sigma e_t,\quad e_t\stackrel{\text{iid}}{\sim} \mathcal{N}(0,1)$$ so the MA(q) model is a "pure" error model, the degree $q$ defining how far the correlation goes back.

• I'm still not clear on where $e_t$ comes from. Is $e_t$ a random variable? I don't think so, otherwise why to bother looking for $q$ correlations? Apr 7, 2012 at 18:31
• Why is there a minus in your formula? Usually the minus is for AR models. Mathematically is not an issue, I'm just curious, since I've never seen minus in MA models. Apr 7, 2012 at 20:10
• @RobertKubrick, are you aware of Wold decomposition theorem? Each stationary process has its corresponding innovation process, that is from where terms $e_t$ come. Apr 7, 2012 at 20:21
• @mpiktas Thanks, that gives some background on the error term, but I am still not clear on where the innovation process comes from, for an innovation to exist there's got to be a forecast somewhere (en.wikipedia.org/wiki/Innovation_(signal_processing)). Is the optimal $Y$ forecast simply $E(Y)$, that is the mean of the series? Apr 7, 2012 at 21:43

See my post here for an explanation of how to understand the disturbance terms in a MA series.

You need different estimation techniques to estimate them. This is because you cannot first get the residuals of a linear regression and then include the lagged residual values as explanatory variables because the MA process uses the residuals of the current regression. In your example you are making two regression equations and using residuals from one into the other. This is not what an MA process is. It cannot be estimated with OLS.

You say "the observation $Y$ is first regressed against its previous values $Y_{t−1},...,Y_{t−n}$ and then one or more $Y−\hat{Y}$ values are used as the error terms for the MA model." What I say is that $Y$ is regressed against two predictor series $e_{t-1}$ and $e_{t−2}$ yielding an error process $e_t$ which will be uncorrelated for all i=3,4,,,,t .We then have two regression coefficients: $\theta_1$ representing the impact of $e_{t-1}$ and $\theta_2$ representing the impact of $e_{t-2}$. Thus $e_t$ is a white noise random series containing n-2 values. Since we have n-2 estimable relationships we start with the assumption that e1 and e2 are equal to 0.0 . Now for any pair of $\theta_1$ and $\theta_2$ we can estimate the t-2 residual values. The combination that yields the smallest error sum of squares would then be the best estimates of $\theta_1$ and $\theta_2$.

• What are the 2 other predictor series? I am asking because when I look at the literature I have it's never clearly specified. Are these 2 other series unrelated to $Y$? I had the impression that all ARIMA formulation is limited to the $Y$ series. Apr 7, 2012 at 20:15
• The 2 predictors are the lags of the error terms. Since these are not known a priori since we do not know the error terms before we begin is why this has to be treated by non-linear estimation.The confusion you are having is that a model that is finite in the past ( i.e. an AR MODEL ) is potentially infinite in the errors AND a model that is finite in the errors ( i.e. an MA MODEL) is potentially infinite in the past of Y.The reason one selects an AR MODEL versus an MA MODEL is for parsimony. Sometimes we construct an ARMA MODEL which blends both the history of Y and the history of the errors. Apr 7, 2012 at 20:59
• As I commented in the other answer, what I am still missing is what's the optimal forecast for $Y$, which is used to calculate the innovation $e_{t-n}$. Apr 7, 2012 at 22:00

With Hannan–Rissanen (1982) algorithm to fit parameters for an ARIMA model you actually always do an AR regression as first step, even for an pure MA model:

1. AR(m) model (with $$m > max(p, q)$$) is fitted to the data
2. Compute error terms for all $$t$$: $$\epsilon_t$$ = $$y_t - \hat{y}_t$$
3. Regress $$y_t$$ on $$y^{(d)}_{t-1},..,y^{(d)}_{t-p},\epsilon_{t-1},...,\epsilon_{t-q}$$ (For a pure MA model the regression would be done only against the error terms $$\epsilon_{t-1},...,\epsilon_{t-q}$$)
4. To improve accurancy optionally regress again with updated model parameters $$\phi,\theta$$ from step 3

So your suspicion that one needs some kind of model first, before one can compute the error part can be converted into an iterative algorithm for fitting the parameters of an ARIMA model.