Is there a simple way to explain the parameter estimation process of MA or ARMA?

Question

I know the ordinary least squares and gradient decent methods that are used to estimate the parameters (weights) in linear regression. But I am not that good at math and I can't understand the optimization process (the algorithm) or the parameter estimation of MA or ARMA.

Answer 1

You come up with the likelihood function $LL(\theta|X)$ which is a function of the set of parameters $\theta$ given the data set $X$. You find the set of parameters $\theta$ that maximizes the function for any given sample $X$. That's simple conceptually. There is no simple way to explain how you come up with the likelihood function.

Here's an example of the simplest MA(1) process, follow the link here. Eq.12 is the model:

$$Y_t=\mu+\varepsilon_t+\theta\varepsilon_{t-1}$$

The likelihood function is based on the assumption that $\varepsilon_t$ is from Gaussian distribution, AND that the sample is random. So, you could re-write this as $$\varepsilon_t=Y_t-(\mu+\theta\varepsilon_{t-1})$$

So, if you plug some arbitrary parameters $\hat\mu,\hat\theta$ and assume $\varepsilon_0=0$ you can estimate the errors recursively as shown on p.15: $$\hat\varepsilon_1=Y_1-\hat\mu$$ $$\hat\varepsilon_2=Y_2-(\hat\mu+\hat\theta\hat\varepsilon_1)$$

If you forget about likelihoods for a moment, you could try to minimize the sum of squares $\sum_t e_t^2(\hat\mu,\hat\theta)$ as a function of its parameters given the dataset $Y_t$. From optimization point of view it's the same problem as maximizing some other nonlinear function $LL$, but otherwise the least squares here is not necessarily a good idea.

The actual likelihood function is in Eq.14, and you can see that it's not that different from least squares in some way. It has negative of sum of squares and some other terms.