4
$\begingroup$

Let's suppose that the considered set of random variable has a covariance matrix which is psd. Therefore the Gaussian pdf must be written in its degenerate form, where the determninat of the covariance matrix is replaced by the pseudodeterminant (i.e. the product of non-zero egeinvalues) and the inverse of the covariance matrix is replaced by the pseudoinverse (Wikipedia link2 to stats.stackexchange).

My question is whether we are still able to make use of the information criteria and tests used to specify the model in the non-degenerate case, like the AIC, BIC, LR Test (Wald test, etc...). More precisely, given a number of observed samples t and a number of parameters k used in the model

$$AIC=-2Loglik+2k$$ $$BIC=-2Loglik+ln(T)2k$$ $$LRstatistic=2(Loglik_{fullmodel}-Loglik_{restrictedmodel})$$

Can these metrics be used in the current form reported above, even when a degenerate multivariate normal distribution used to compute the Loglik? For example, does the LRstatistic in this case preserve its Chi-sqaured distribution as for the standard non-degenerate case?

Clearly, a good hint on the way to solve the problem is to analyze the empirical distribution of LR test statistic using a simulation and the derivation of Akaike's and other ICs to see whether the non-singularity of the varcov matrix is strictly necessary to the derivation of the ICs. As far as the latter point is concerned, I have checked the derivation of Akaike's IC (which is also available here) and, in my opinion, I do not see that the non-singularity of the varcov matrix is strictly necessary. But I would like to hear the forum opinion on the point.

EDIT: notice that here we are talking about a probability model and the support of the varcov matrix is potentially allowed to vary across different specifications to be found with ICs/LR test

$\endgroup$
8
  • 2
    $\begingroup$ Please provide some context for understanding this question: exactly what do you mean by "the considered set of random variables" and what is your "covariance matrix"? For instance, are you supposing some kind of probability model with a degenerate covariance matrix, or is it perhaps the case that the estimated covariance matrix for a set of estimates is nondegenerate? And in the first case, is the support of the covariance always the same subspace for every instance of the model or does the support vary? $\endgroup$
    – whuber
    Commented Aug 2, 2019 at 14:42
  • $\begingroup$ @whuber I am supposing a probability model and allowing the support to vary across different possible model specifications (I upvoted because the question was useful) $\endgroup$
    – Fr1
    Commented Aug 2, 2019 at 14:45
  • $\begingroup$ Can you maybe write down the model and it's implied covariance matrix and elaborate a bit on the reason for the degeneracy? $\endgroup$ Commented Aug 2, 2019 at 16:03
  • 1
    $\begingroup$ @StoryTeller0815 yes also in my context usually we use constraints for a pd varcov, but on this occasion we cannot set constraints because, for reasons a bit long to explain here, the residuals must by definition have a psd varcov. I read that going Bayesian is a good way to overcome the issue. However, working both analytically (i.e. assuming my varcov is any varcov and solving with symbols of each scalar in the varcov) and numerically, I solved the problem reaching the expected solution. So I was wondering, why not staying in the degenerate MLE case? Especially if it worked numerically. $\endgroup$
    – Fr1
    Commented Aug 2, 2019 at 17:11
  • 1
    $\begingroup$ For the numerical resolution I used Python Scipy: indeed in the docs they state that if the varcov is singular then the formulas used will be automatically updated to the degenerate case in order to allow a numerical resolution docs.scipy.org/doc/scipy-0.14.0/reference/generated/… and I must admit that the numerical solution of the optimization converged exactly where we were expecting for all the experiments and simulations $\endgroup$
    – Fr1
    Commented Aug 2, 2019 at 17:12

1 Answer 1

1
$\begingroup$

At least for AIC/BIC, I am very sure, that these measures do not care at all as long as your likelihood function is valid.

Based on the most extensive resources on information theoretic criteria, I believe that it does not matter at all. Of course, AIC/BIC may be "rescaled" if you change the likelihood function and hence, you should not compare these AICs to AICs from models with other likelihood functions. But apart from that, I don't see why they should become invalid as they are just approximations of the distance between the model and reality.

References: Multimodel Inference: Understanding AIC and BIC in Model Selection Kenneth P. Burnham and David R. Anderson Sociological Methods Research 2004; 33; 261 DOI: 10.1177/0049124104268644

$\endgroup$
3
  • $\begingroup$ thank you great answer, great sources, upvoted. Can you please tell me what do you mean exactly by “rescaled” when you say “AIC/BIC must be rescaled when you change the likelihood function”? $\endgroup$
    – Fr1
    Commented Aug 2, 2019 at 17:55
  • $\begingroup$ You're welcome. Per se the absolute value of the likelihood (and therefore AIC/BIC) is meaningless. Only differences matter because these have a scale (this is explained in detail in the second reference). If you use a different likelihood between two models, you may run into trouble. A famous case of this occurs in the context of the linear mixed model. If you use REML estimation, the likelihood is rescaled whenever you change your fixed effects (because these are essentially partialed out) - hence, you may not use AIC/BIC to compare REML models with differing fixed effects. $\endgroup$ Commented Aug 2, 2019 at 18:05
  • $\begingroup$ The same may occur here, when you try to compare a degenerate normal distribution likelihood model and the traditional normal distribution likelihood model by AIC/BIC. I don't know if this happens. I just wanted to warn you that it might happen. $\endgroup$ Commented Aug 2, 2019 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.