I need to perform some research on the consequence of a singular Fisher Information matrix in statistical inference.

I am confused what kind of problems a singular Fisher Information matrix creates. I've recieved a paper 'Likelihood-based inference with singular information matrix' by Rotnitzky (2000) but this seems to hard for me. I don't even get past the motivating example.

I also got a paper on the skew-normal distribution where the information matrix would be singular. The paper 'Problems of inference for Azzalini’s skew-normal distribution' by Pewsey is easier to comprehend. I've tried implementing this likelihood for the so called 'direct' parameterization where $X$ is skew-normal distributed if it's pdf is given by: $$f_X(x;\xi, \omega,\alpha) = \dfrac{2}{\omega}\cdot \phi\left(\dfrac{x-\xi}{\omega}\right) \cdot \Phi\left(\alpha \cdot\dfrac{x-\xi}{\omega}\right)$$ Where $\phi$ and $\Phi$ are the pdf and cdf of a standard-normal.

Using the package sn i've implemented this as follows:


## likelihood function
skewnorm.likl <- function(theta, y){
  xi <- theta[1]
  omega <- theta[2]
  alpha <- theta[3]
  n <- length(y)
  logl <- n*log(2) +n*log(omega)+ sum(log(dnorm((y-xi)/omega))) + sum(log(pnorm(alpha*(y-xi)/omega)))

## testing it out
## this generates 1000 skew-normal distributions
y <- rsn(n=1000,xi=2,omega=2,alpha=10)

## trying the optimization
optim(c(0,1,5), skewnorm.likl, y=y)

Finding the MLE by minimizing the loglikelihood seems impossible trough calculations, these methods don't converge.


Could someone clarify the issue of a singular Fisher information matrix, through the implementation in R it seems like the issue is that finding the MLE is hard (by optimization).

On the other hand this doesn't really adress the issue of inference... I also know how: $$\sqrt{n}(\boldsymbol{\hat \theta} - \boldsymbol{\theta}) \stackrel{D}{\rightarrow} N\left( \boldsymbol{0}, I(\boldsymbol{\theta})^{-1}\right)$$ which also seems like an issue of $I$ is singular...


The goal of the Rotnitzky et al. (2000) paper is (quote)

...to provide a unified theory for deriving the asymptotic distribution of the MLE and of the likelihood ratio test statistic when the information matrix has rank one less than full and the likelihood is differentiable up to a specific order.

Their results depend on a parameter $s$ and its parity (odd/even), where $2s+1$ is the number of derivatives of the likelihood. In such a situation, what happens is that depending on the parity of $s$, the statistic that has an asymptotic distribution is not the usual one (has different scaling), and the limiting distribution is not normal. See their Theorems 1 and 2 in subsection 3.4, and their comment immediately below them.

For the Skew-normal distribution, the Fisher Information becomes singular when the true value of the "skew" (or "slant") parameter $\alpha$ is zero. This means, as Azzalini notes in ch. 3.1, pp 57-59, that the score test with the null hypothesis $\alpha =0$ is invalid in its usual form.

When $\alpha =0$ two of the elements of the gradient of the log-likelihood become linearly dependent, which creates the singularity.

  • 1
    $\begingroup$ So if I understand well: The main issue of a a singular Fisher Information matrix is that it implies that the usual $$\sqrt{n}(\hat \theta-\theta) \stackrel{D}{\rightarrow} N(0, I(\theta)^{-1})$$ is not valid. Rotnitzky et all gives the valid asymptotic distributions. This is important since the likelihood ratio test uses the asymptotic distribution. The fact that calculating the mle estimators through optimization is harder when $I$ is singular is just another phenomenon. $\endgroup$
    – dietervdf
    Apr 12 '17 at 19:29
  • $\begingroup$ Indeed, since its inverse stands for the asymptotic variance-covariance matrix, when this inverse does not exist, the asymptotic distribution should be expected to somehow change. $\endgroup$ Apr 12 '17 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.