3
$\begingroup$

I have some trouble understanding the forecasting/inference process of ARMA models.

From Hamilton (which I am reading now), we can obtain forecasts at $Y$ from any linear process with r.v. values $X$ using the expectations as follows:

$\mathbf{a} = E(XX^T)^{-1}E(XY^T)$

This assumes that the forecast is the optimal linear forecast using projections. The matrix $E(XX^T)$ is the matrix of autocovariances (since ARMA is stationary), and $E(XY^T)$ is also a matrix of autocovariances.

1) We can establish all autocovariances from the sample (time series) and forecast the future values using the above formula by simply substituting estimates for second moments, inverting matrices, and we are done. Why do we need to find the ARMA coefficients to forecast which is done in all packages like stat::arima?

2) I still cannot figure the whole process - stat::arima uses the Kalman filter to compute the fundamental innovations, and somehow compute the likelihood (cannot decipher Hamilton chapter 13 yet since have not yet read the prerequisite chapters). From stat::arima:

The exact likelihood is computed via a state-space representation of the ARIMA process, and the innovations and their variance found by a Kalman filter.

So, these innovations + observations should be enough to forecast the future values. Why do we even need likelihood at all? From Gardner et al. https://www.jstor.org/stable/2346910:

The prediction and updating are carried out by means of a set of recursive equations known as the "Kalman filter" The parameter $\sigma^2$ does not appear in the recursion

3) When Kalman filter finds the innovations - it finds also the variance of those and computes the maximum likelihood value. BUT HOW ARE PARAMETERS of ARMA for maximum likelihood found? Kalman can compute the maximum likelihood values, but how are the maximum likelihood parameters $\theta$ and $\phi$ selected? As far as I know, to find these parameters, some gradient descent has to be used, or Newton method. From Gardner again:

The log-likelihood function may then be maximized with respect to ($\theta, \phi$) by minimizing:

$\mathcal{L}(\theta, \phi) = n \log S(\theta, \phi) + \sum_{i = 1}^n \log f_i, \quad \quad f_i \propto \hat{MSE}$

So, the procedure described in the paper outputs the values of errors. likelihood, etc. But where does it get the optimal $\theta, \phi$?

Improve this question
$\endgroup$
  • 1
    $\begingroup$ Why do we even need likelihood at all? Innovations and observations are not enough, the parameters of the model ($\phi\,,\theta$) needs to be estimated. ARMA models with the same orders, eg. ARMA(2,2), but with different values of the AR and MA coefficients do not involve the same features (period of cycles, persistency,...). This is where the likelihood function comes into play; it measures the likelihood or plausibility of a set of parameters values given the observed data. So the goal is to chose those parameter values that maximise the likelihood function. $\endgroup$ – javlacalle Jan 4 '19 at 20:32
  • 1
    $\begingroup$ But how are parameters of ARMA for maximum likelihood found? The likelihood function is maximised with respect to the parameters of the model (AR and MA coefficients). To do so, an optimisation algorithm and the Kalman filter can be combined as follows: The Kalman filter is used to compute the value of the likelihood function for a given set of parameters values; it is passed as argument to an optimisation algorithm, eg. Newton method. As a result, a set of parameter values that maximises the likelihood function are obtained and used to compute forecasts upon the given model. $\endgroup$ – javlacalle Jan 4 '19 at 20:36
  • 1
    $\begingroup$ But why not estimate [second moments] from data? Calculating the sample autocovariances from the data is a natural alternative. In fact, the theory of ARMA models establishes a mapping between the theoretical autocovariances and the parameters of the model. This approach is known as the method of moments. For AR models it amounts to the Yule-Walker system of equations. When MA is present, the system of equations is less simple to solve. $\endgroup$ – javlacalle Jan 4 '19 at 22:41
  • 1
    $\begingroup$ Maximum likelihood has in general better properties than the method of moments, related post. The method of moments is nonetheless a helpful procedure to get initial parameter estimates from which to start the search of ML estimates. $\endgroup$ – javlacalle Jan 4 '19 at 22:42
  • 1
    $\begingroup$ Once the algorithm that combines the Kalman filter and the Newton method converges to a set of parameter values, the data can be extrapolated by means of the Kalman filter (maybe this is what you see related to your last comment). In practice, it may be better to use the Kalman smoother to obtain forecasts (employs information on the whole data set). $\endgroup$ – javlacalle Jan 4 '19 at 22:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.