I am trying to estimate the standard errors of an maximum likelihood estimate (multidimensional) in R'sfunction optim. I want to this by the observed information matrix. Since I minimize the negative log-likelihood $-\log \mathcal{L}$, I can find the standard errors by the diagonal elements of the inverted Hessian matrix at the optimal solution, which is found by the argument hessian = TRUE in optim.

The function does however return a matrix which is singular: solve("Hessian") returns the error ...system is exactly singular: U[44,44] = 0.

The model is a bivariate mixture model where the proportions of the components are determined by a logistic regression. Also, the shape of the distribution is also affected by some set of covariates. All covariates are categorical/dummy variables. T

he regressions within the model has an intercept, corresponding to a base case from which all other dummy variables are compared. By that I mean, for 3 levels, A, B and C, the model takes the form $\beta_0 + \beta_B x_B + \beta_C x_C$.

So my question is what this means and how can we remedy/handle this? Does anyone know?

edit: I awkwardly overlooked an issue yesterday that I realized is the root of the problem. I'm not enitrely sure how to handle it though. In the model, I use a logistic regression to find the probability $p$ of one component (the other being $1-p$). For one covariate, this probability goes clearly seem to "converge" to 1. By that I mean the corresponding coefficient estimate is very large (~30-35). So the we have $\frac{e^{35}}{1+e^{35}}$ so this is basically a vertical line at 1 w.r.t. to that coefficient (parameter), yielding a 0 in that diagonal element of the hessian matrix, i.e. the second derivative is 0.

For these kind of problems, can I omit that variable (exclude the row and line corresponding to that parameter) of the hessian and correctly compute the standard errors of the other variables by inverting the remaining matrix.

  • $\begingroup$ Can you post more of your code? It would help to see the full form of your model and the optim output. A singular matrix can be due to error in coding or highly dependant covariates. $\endgroup$ – Zachary Blumenfeld May 21 '15 at 16:13
  • $\begingroup$ You were estimating a finite mixture model. Did you use the expectation-maximization (EM) algorithm or simply optim the log likelihood function? If you directly optim the log likelihood, the optimization could encounter the singularity problem, as well known for the gaussian mixture models. You can consider use the R package mixtools to implement the EM algorithm. I think the covariance matrix there is produced by bootstrap. $\endgroup$ – semibruin May 21 '15 at 16:14
  • $\begingroup$ Unfortunately I do not have the code on this computer. I'm pretty certain the code is fine though. Yes I use the EM algorithm, I do however specify the observed likelihood function for which I estimate the Hessian matrix. I realized what was causing the error, not entirely sure how to handle it though. I updated my question. $\endgroup$ – Good Guy Mike May 22 '15 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.