Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm writing Python code to use the Kalman State-space approach to estimate ARMA model coefficients using MLE however, I'm not too clear on how to derive the coefficient estimates standard errors from the process.

I know you have to use the information matrix, invert it to get the variance-covariance matrix and you can derive it from there, but how to derive the information matrix is the challenge I'm currently faced with (from a coding standpoint).

Any links to free papers, tutorials, equations, or references to code is what would be very helpful to me.

Thanks!

share|improve this question
1  
You could always download R and look at the source code of arima() for inspiration. –  Chase Jul 17 '11 at 22:28
add comment

2 Answers 2

Usually the by-product of optimization algorithms is hessian of optimisation function. If optimisation function is the log-likelihood then the hessian returned will be the estimate of the information matrix. Here is the relevant formula from wikipedia page:

$$\mathcal{I}(\theta)=-E\frac{\partial^2 \log f(X)}{\partial \theta \partial \theta '}=E\frac{\partial \log f(X)}{\partial \theta}\frac{\partial \log f(X)}{\partial \theta'}$$

The log-likelihood of the sample supplied to the optimisation function is of the form:

$$l(\theta)=\frac{1}{n}\sum_{i=1}^n\log f(X_i,\theta)$$

Hence the hessian of this function estimated on the optimal point is:

$$\frac{\partial^2 l(\theta)}{\partial\theta\partial\theta'}=\frac{1}{n}\sum_{i=1}^n\frac{\partial^2\log f(X_i,\theta)}{\partial \theta\partial\theta'},$$

so we estimate the expectation with a sample average, which will tend to the true expectation due to law of large numbers. If you look at the arima code as @Chase suggested, you will see that this is exactly the way R calculates the variance matrix of the coefficients (as the inverse of hessian, which is the information matrix).

If the optimisation algorithm does not return the hessian, you can exploit this formula for information matrix:

$$ \mathcal{I}(\theta)=E\frac{\partial \log f(X)}{\partial \theta}\frac{\partial \log f(X)}{\partial \theta'} $$

The estimate using the gradients will be then

$$\frac{1}{n}\sum_{i=1}^n\frac{\partial \log f(X_i)}{\partial \theta}\left(\frac{\partial \log f(X_i)}{\partial \theta}\right)'.$$

But for this you will need somehow to estimate the gradient at each point $X_i$. This can be done either numerically or analytically. Note that since the differentiation is done with respect to $\theta$, with $X_i$ as constant it might not be that hard.

share|improve this answer
add comment

I have no experience with such models. However, there is a time varying copula model of Patton (2006) that is estimating a dependence between two series and this dependence is non-linear (so, he is using MLE) and follows an ARMA-type process. There he derives the standard errors for the estimates. There is also a reference for the literature provided in his paper. On his website, he has the codes to model that. However, I do not remember, if the codes for the standard errors are there.

Here is the paper http://econ.duke.edu/~ap172/2stage_published_version_mar06.pdf check p.152 His codes for this paper are here http://econ.duke.edu/~ap172/code.html

Hope, this information is a little relevant to you problem. Good luck.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.