I would like to compute the Fisher information of an Ornstein-Uhlenbeck process $X_t = Y_t - \beta Z_t$ where $Y_t$ and $Z_t$ are two time-series.

My log-likelihood function in this case is:

$$\mathcal{L}(\mu, \lambda, \hat{\sigma}, \beta)=\sum_{i=1}^{n} \log f\left(X_{i} X_{i-1} ; \mu, \lambda, \beta, \sigma\right)$$ $$=-\frac{n}{2} \log (2 \pi)-n \log (\hat{\sigma}) -\frac{1}{2 \hat{\sigma}^{2}} \sum_{i=1}^{n}\left[(Y_{i} - \beta Z_{i})- (Y_{t-1} - \beta Z_{t-1}) e^{-\lambda \delta}-\mu\left(1-e^{-\lambda \delta}\right)\right]^{2}.$$


$$ \hat{\sigma}^{2}=\sigma^{2} \frac{1-e^{-2 \lambda \delta}}{2 \lambda}. $$

The Fisher information is defined as:

$$ \mathcal{I}(\theta)=\mathrm{E}\left[\left(\frac{\partial}{\partial \theta} \log f(X ; \theta)\right)^{2} \mid \theta\right]=\int\left(\frac{\partial}{\partial \theta} \log f(x ; \theta)\right)^{2} f(x ; \theta) d x $$

or equivalently (if the log-likelihood is twice differentiable):

$$ \mathcal{I}(\theta)=-\mathrm{E}\left[\frac{\partial^{2}}{\partial \theta^{2}} \log f(X ; \theta) \mid \theta\right] $$

where $\theta$ is a vector of my four parameters $\mu, \lambda, \hat{\sigma}, \beta$.


Question: Obviously I have to take partial derivatives w.r.t to each parameter, once or twice depending which version of Fisher information I use. However the computation easily becomes massive, where partial derivatives are extremely long.

What is the right way to do this since it does not seem feasible to take partial derivatives by hand here?

Thank you

  • $\begingroup$ Do you need analytical formulas? If not, eg you are implementing this for computational methods, you may want to use auto differentiation packages $\endgroup$ – Cam.Davidson.Pilon Apr 18 at 21:16
  • $\begingroup$ @Cam.Davidson.Pilon I did want to write a short document with the formulas and tried using Sympy but its output is a bit all over the place. Do you have any recommendations for the auto differentiation packages, even if only for computational purposes rather than analytical formula? $\endgroup$ – MilTom Apr 18 at 21:34
  • $\begingroup$ I like Python's autograd - very light weight and easy to get started. w.r.t. the document with formulas, here are some questions to think about - do you think analytical formulas, as messy as they are, are doing to be useful to the reader? What do you expect the reader to do with them? $\endgroup$ – Cam.Davidson.Pilon Apr 19 at 0:18
  • $\begingroup$ As I understand your model is for the response $Y_t$; $Z_t$ is a covariate and $X_t$ is the OU noise. You can put this model in state-space (SS) form allowing to filter, smooth and predict. The log-likelihood writes as the sum of an initial term and a sum of squares of the innovations. By differentiating the SS matrices w.r.t. to the parameters you can differentiate each term in the log-likelihood. The computations are made in linear time. See Sec. 3.4 of Andrew Harvey's book Forecasting, Structural Time Series Models and the Kalman Filter. $\endgroup$ – Yves May 3 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.