In maximum likelihood estimation (MLE) a useful result is that the standard errors for some estimated coefficient vector can be computed as the square roots of the diagonal entries of the inverse of the negative expected information matrix.

In other words, let the empirical estimate of the expectation of the information matrix be: $$I(\theta) =-\frac{1}{N}\sum_{i=1}^{N}\mathcal{H}(Z_{i}, \theta)$$ for parameter vector $\theta$, and data points $Z_{i}$, where $\mathcal{H}(Z_{i},\theta)$ denotes the Hessian matrix of the log-likelihood function, evaluated at the single datum $Z_{i}$ and the parameter vector estimate $\theta$.

Then the matrix given by $I(\theta)^{-1}$ contains the variance/covariance structure of the MLE estimates, and its diagonal entries in particular are the variances of the estimated MLE parameters.

In my specific application, the $Z_{i}$ are consumer data about price preferences for different heating options, and the model is a logit probability model. I've done all the work of estimating the MLE and there are standard formulas for the derivative vector and Hessian matrix in this logit scenario. I have about 250 data samples, which is large enough that I expect the MLE estimate to be fairly accurate.

However, when I numerically compute the information matrix and the corresponding variances, I am seeing very large numbers. The estimated coefficients range in absolute value from 0.5 to 22, but the smallest standard error is about 7.2, which makes me question what I'm doing.

I know that the empirical estimate of the Hessians are correct, as I've used my code on a separate data set to confirm that I'm computing the derivatives and second-derivatives correctly, so I think it's unlikely to be a coding mistake.

Does anyone have any advice on what might yield such large standard deviations in practice?

  • $\begingroup$ Why do you expect the MLE to be more accurate than this? A sample size of 250 isn't really that high for a logit model, especially if the probabilities are low so the number of 'success' outcomes is fairly small. $\endgroup$ – onestop Feb 25 '12 at 21:19
  • $\begingroup$ It's a multi-class logit model, so there are 5 choices the consumers could make. Are you saying that for choices that are barely encountered in the data set, I should expect such high standard errors? It just seemed suspicious to me, but then again, it's hard to know what 'large' means in the MLE parameter space. I can have a look at the data and see what the relative frequencies are... $\endgroup$ – ely Feb 25 '12 at 21:25
  • 2
    $\begingroup$ If it's not small sample size, as noted above, it could be due to collinearity of predictors or non-identifiability of regression coefficients. $\endgroup$ – Macro Feb 26 '12 at 0:42
  • $\begingroup$ @Marco, I think multi-collinearity is a good explanation after reading about that. If you feel like summarizing that concept and maybe throwing in a reference to how it specifically relates to MLE (as opposed to standard regression), I'd be happy to accept it as an answer. $\endgroup$ – ely Feb 26 '12 at 3:35
  • $\begingroup$ As an update, I've run into the same relative issue with a much larger data set, 2500 samples, and so I'm not sure that collinearity is the whole story. In this new estimation, I'm looking for price sensitivity parameters for purchase choices, so not really a regression kind of problem. I'm guessing it has something to do with the MLE's implicit use of an improper prior, which inflates the true uncertainty in the estimates. $\endgroup$ – ely Mar 4 '12 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.