# Estimating ARCH model using ML or OLS

ARCH(p) models are defined as:

$σ^2 = a_0 + ∑a_ie^2_{t-i}+e_t$, $i>0$

Now, as with any VMA model, estimating this model using OLS/ML is impossible, because the error term is not observable. But there still appears to be a solution, which is to first estimate the AR(p) model:

$Y_t = ∑B_iX_{t-i} +e_t$

and afterwards use the resulting error terms to estimate the arch model:

$e^2_t = a_0 + ∑a_ie^2_{t-i} + e_t$, $i>0$

Where does the equivalence of $σ^2$ and $e^2_t$ come from? I thought that the variance is in the first model was simply the value of depended value squared, but apparently not. Also if the σ is indeed the residual of an AR(p) model, then the $e^2_t$ entirely depends on the lag selection of the said AR(p) model. How do the automated package generally estimate ARCHs given that fact? Do they use same amount of lags for both equations? In addition, how would you go about adding GARCH components using this method?

It seems that you did not get the equations (nor the idea) quite right.

The formula for the AR($p$) model should be

$$y_t=\beta_0+\beta_1 y_{t-1}+\dotsb+\beta_p y_{t-p}+\varepsilon_t;$$

the formula for the ARCH($s$) model should be

$$\sigma^2_t=\omega+\alpha_1 \varepsilon^2_{t-1}+\dotsb+\alpha_s \varepsilon^2_{t-s}.$$

(I used $\omega$ where you used $\alpha_0$ but this makes no difference.)

We can estimate an AR($p$) model consistently even if we neglect the ARCH structure in the model error -- because AR($p$) can be expressed as a linear regression (although there are $p$ missing observations for lagged series), where conditional heteroskedasticity leads to inefficiency but does not prevent consistency.
Errors $e_t$ obtained from such a model will be consistent estimates of the true underlying disturbances $\varepsilon_t$ as long as the AR($p$) model is the correct model for the conditional mean of the process.
Then the errors $e_t$ can be used in place of the true underlying disturbances $\varepsilon_t$ to estimate the ARCH$(s)$ model (usually via maximum likelihood):

$$\sigma^2_t=\omega+\alpha_1 e^2_{t-1}+\dotsb+\alpha_s e^2_{t-s}$$

Note that there is no error term on the right hand side (since $\varepsilon$ is treated as a variable, not an error in the ARCH model). The ARCH model assumes a perfect fit for the conditional variance. That is, the conditional variance is assumed to be deterministic -- not stochastic. This is in contrast to stochastic volatility models.

Now, as with any VMA model, estimating this model using OLS/ML is impossible, because the error term is not observable.

This is not really true. Although OLS cannot work directly with latent variables, ML can. VMA can be estimated via ML.

Where does the equivalence of $\sigma^2$ and $e^2_t$ come from?

There is no equivalence between $\sigma^2_t$ and $\varepsilon^2_t$. Given that, some of your further questions become irrelevant.

Do they use same amount of lags for both equations?

The lag order of the AR($p$) model and the ARCH($s$) model need not be the same. Selecting the best lag orders may be tough in practice because the models are interdependent. (The AR($p$) model may be estimated first and ARCH($s$) model estimated subsequently, but the efficient way is to estimate both models simultaneously.) One option is to use a grid of $p$ and $s$ values, estimate all models over the grid and pick the model with the lowest AIC or BIC value. However, this approach is subject to overfitting, especially if the grid is fine.

• One quick (dumb) question. If the conditional variance is indeed deterministic, why does the Arch(S) model (equation 2 of your answer) need any estimation via ML at all? Can't I just input the values and have the variances? Or if I am still understanding this wrong and the w and a parameters are to be estimated (and not brought over from the equation 2), then where do I find the values for sigma squared? – Dole Jun 9 '15 at 16:27
• Edit to the above question. I recon that the ARCH equation to be estimated is actually the third one from my original question (the last error term should be V_t however). As a result the variance variable moves to the error term. Then the resulting coefficients can be inputted into the 3rd equation in your post. Maybe some program can estimate it outright without the form change. – Dole Jun 9 '15 at 16:47
• In the ARCH model, $\sigma^2_t$ are unobserved while model parameters $\omega$ and $\alpha$'s are unknown, so there is no easy way to just input the values. During the estimation of an ARCH model the $\sigma^2_t$'s are estimated together with the model parameters. (Otherwise it could be difficult to get the perfect fit assumed by the model.) In general, $\sigma^2_t$ (or $\sigma^2$) are never observed, there can only be better or worse proxies for them. – Richard Hardy Jun 9 '15 at 17:04
• Still having a comprehension problem, how can an equation with no error term be estimated/perfect fit be achieved, unless there are t parameters? Here also is a quick step by step guide guide for those interested: blacklen.wordpress.com/2010/09/22/…. It suggests that the variance term can be had by dividing the error term by white noise and squaring if I am not mistaken. – Dole Jun 10 '15 at 1:09
• Perfect fit is relatively easy here here because were have more "free parameters" (the term includes both the latent $\sigma$'s and the ARCH model parameters $\omega$ and $\alpha$'s) than there are data points. Regarding the other question, I repeat that the variance is unobserved, so whatever is there to proxy for it is just a proxy... – Richard Hardy Jun 10 '15 at 6:10

I really felt i needed to clarify this question even though it is fairly old, because there are a lot of misconceptions in this thread.

The MLE (Maximum likelihood estimator) & QMLE (Quasi Maximum Likelihood Estimator) of the ARCH(q) model is both consistent and asymptotically normal for specific ranges of the parameters depending on the size of q. There are less parameters than data points provided that the available dataset is Larger than the amount of parameters sought estimating. This is very rarely a problem, since the prime use of the ARCH model is financial time series, in which there are often several thousand data points. So the amount of degrees of freedom used for estimation will rarely be a problem. The laglength, q, is NOT defined by the amount of data.

To completely clarify this, ill write up the ARCH(1) model in a similar way to what Robert Engle did in his seminal work on the ARCH model back in 1982.

The ARCH model is repressented by the following 3 equations:

$$(1)\,\,y_{t}=\delta+\epsilon_{t}$$

$$(2)\,\,\epsilon_{t}=\sigma_{t}z_{t},\,\,z_{t}\sim N(0,1)$$

$$(3)\,\sigma_{t}^{2} =\omega+\alpha\epsilon_{t-1}^{2}$$

Further it is assumed that $$z_{t}$$ is independent across all t, and that $$\alpha_{1} \geq 0$$ and $$\omega>0$$. The two last assumptions ensure that $$\sigma_{t}^2$$ is always positive.

The Classical ARCH model only operates with equation (2) and (3), equation (1) is altered a bit such that we may look at $$\epsilon_{t}$$ as the demeaned return of some stock. The reason for this specification can be thought of as the assumption that stoch markeds are atleast weakly efficient, which tends to be true in empirical studies. Alleviating the assumption of weak efficiency one could introducing p lags of $$y_{t}$$ to equation (1). This will turn the model into the AR(p)-ARCH(1) model, which is a whole other model with different requirements for stability and thus estimator properties - so i will omit these as they are much more complicated to work with (although this is rarely mentioned in introductory ARCH/GARCH courses).

The first thing to clarify is that equation (2) is by all means stochastic which verifies that the model does have a residual. $$\epsilon_{t}$$ is not known given the information set $$I_{t-1}=\{y_{0},y_{1},...,y_{t-1}\}$$ however the function for the conditional variance, $$\sigma_{t}^2$$ is. This is what they mean by the fact that $$\sigma_{t}^2$$ is deterministic, or rather constant conditional on $$I_{t-1}$$ in opposition to the stochastic volatility models (see for example the log normal stochastic volatility model).

In order to make sense of this model, and how to estimate it by MLE we have to know the conditional distribution for $$y_{t}$$ given $$I_{t-1}$$, such that we can write the conditional likelihood function. If $$z_{t}$$ is not normal but a more heavy tailed process we may still, in some cases, think of it as normally distributed during estimation to obtain a well behaved QMLE.

To obtain the conditional distribution we first derive the conditonal mean and variance of $$y_{t}$$.

We start by the conditional mean of (1) first:

$$E(y_{t}|I_{t-1})=E(\delta+\epsilon_{t}|I_{t-1})=\delta + E(\sigma_{t}z_{t}|I_{t-1})$$

Because $$\sigma_{t}$$ can be calculated by through $$I_{t-1}$$, and the independence assumption in regard to $$z_{t}$$ implies that $$E(z_{t}|I_{t-1})=E(z_{t})$$ we obtain:

$$E(y_{t}|I_{t-1})=\delta+\sigma_{t}E(z_{t})=\delta$$

Thus we have derived the conditional mean.

Now we derive the conditional variance:

$$Var(y_{t}|I_{t-1})=E((y_{t}-E(y_{t}|I_{t-1}))^2|I_{t-1})=E((y_{t}-\delta)^2|I_{t-1})=E(\epsilon_{t}^2|I_{t-1})$$

We substitute in equation (2) and obtain:

$$Var(y_{t}|I_{t-1})=E(\sigma_{t}^2z_{t}^2|I_{t-1})=\sigma_{t}^2E(z_{t}^2|I_{t-1})$$

In the last equality we used that $$\sigma_{t}$$ is parfectly defined by the informationset, and that $$z_{t}$$ is independent of itself across time. Using that $$z_{t}\sim N(0,1)$$ we get

$$Var(y_{t}|I_{t-1})=\sigma_{t}^2$$

By abusing normality of $$z_{t}$$, namely the rule that for any normal distributed variable $$q\sim N(m,v)$$ and constants $$(a,b)\in\mathbb{R}^{2}$$ it holds that the linear combination given by $$l=a+bq$$ follows a normal distribution given by $$N(bm+a,vb^{2})$$.

We obtain that:

$$y_{t}|I_{t-1}\sim N(\delta,\sigma_{t}^2)$$.

This gives us the conditional likelihood function given by:

$$L(\theta|y_{0})=\prod_{t=1}^{T}\frac{1}{\sqrt{2\pi\sigma_{t}^{2}}}\exp\left\{ -\frac{1}{2}\frac{(y_{t}-\delta)^{2}}{\sigma_{t}^{2}}\right\}$$

Where the number T is the total number of observations excluding the initial one, which we condition on, and $$\theta=(\delta,\omega,\alpha_{1})$$ is our parameter vector we seek to estimate.

The log-likelihood function is given by:

$$\mathcal{L}(\theta|y_{0})=-\frac{1}{2}\sum_{i=1}^{T}\left[\ln(2\pi)+\ln(\sigma_{t}^{2})+\frac{(y_{t}-\delta)^{2}}{\sigma_{t}^{2}}\right]$$

And finally the maximum likelihood estimator is given by:

$$\hat{\theta}_{ML}=\arg\max_{\theta}\mathcal{L}(\theta|y_{0})$$

By definition of the likelihood problem we have 3 equations given by the First order conditions (FOC) and 3 parameters in $$\theta$$. Because of this and the functional form of equations (1), (2) and (3) each of the estimates are identified by the FOC's. So there is no problems with estimation arising from this either.

To close the final bit of confusion that seems to have appeared here. We run into the problem that the FOC representing the maximization problem does not have an analytical solution. That is there is no way to solve the equation system by standard math - or rather no formula. This does not mean that no solution exists. That is by using numerical optimization such as BFGS, Newton-Rahpson or Nelder mead one can obtain a solution to the 3x3 equation system represented by this particular problem.

For this particular problem, the MLE estimator is consistent and asymptotically normal for the estimator of $$\alpha_{1}$$ and $$\omega$$ as long as $$y_{t}$$ is strictly stationary. This holds when $$\alpha_{1}$$ is smaller than approximately 3.56. See Daniel B. Nelson (1990) "STATIONARITY AND PERSISTENCE IN THE GARCH(1,1) MODEL" Econometric theory, 6, nr 3, 318-334 as well as Jensen & Rahbek (2004) "ASYMPTOTIC NORMALITY OF THE QMLE ESTIMATOR OF ARCH IN THE NONSTATIONARY CASE" Econometrica, 72, nr. 2, 641-646. Further the last paper shows that the MLE estimator of $$\alpha_{1}$$ is consistent and asymptotically normal for all values of $$\alpha_{1}$$ which is a very strong result. This suggests that verifying the behaviour of the other parameter estimates are simple, since one just need to test of $$\alpha_{1}<3.56$$. This is a property only known to hold for the MLE and QMLE, and is the reason why parameters in ARCH models are almost never estimated with other estimators.

I would conjecture however that the estimator of $$\delta$$ is only asymptoticaly normal given that $$\alpha_{1}<1$$, and consistent if $$\alpha_{1}<\pi/2$$. This is because $$\alpha_{1}>1$$ ensures that $$E(y_{t}^2)=\infty$$ mich messes up the CLT, and $$\alpha_{1}>\pi/2$$ ensures that $$E(y_{t})$$ is undefined - messing up the LLN for ergodic and stationary series. However, this might not cause problems, as it does not do so for estimation of $$\alpha_{1}$$ and $$\omega$$.

To verify this, one can analyze when the FOC of the MLE satisfies lemma 2 in Jensen & Rahbek (2004b) (Asymptotic inference for nonstationary GARCH.). This is a slightly more complicated endevour, and i will leave that to the reader to verify this if needed be.

• Welcome to Cross Validated and thank you for a very nice answer! There are very few people on Cross Validated contributing to threads on GARCH models, so it would be a delight to receive more of your contributions in the future, too. (When it comes to understanding GARCH models, I have come a long way myself over the course of the last few years; I see that my early answers were less than perfect.) – Richard Hardy May 22 at 13:00