1
$\begingroup$

I am interested in comparing whether a lognormal or a power law are a better fit for a given set of data. Both distributions have been fit using MLE, with $x_{min}$ determined using KS-minimization a la Clauset et al. (2009).

I want to use the AIC to do the comparison of the two fits, but have seen warnings oon the AIC wikipedia article and here, not to use the AIC when one of the variables is log-transformed and the other isn't. In the PL we don't really have log transform to fit the variable, but for the LN we are, so is computing the AIC's and comparing them not OK to do? They are quite quite similar.

On the other hand some have pointed out here that the AIC is only for nested models, while the Wikipedia article on AIC cites Burham and Anderson (2002) stating the AIC is valid for non-nested models and in fact has the advantage over likelihood ratio tests of not requiring models to be nested. The PL and LN are not nested models as I understand.

This is all in the context of using AIC instead of the likelihood-ratio (L-ratio) test suggested by Clauset et al. (2009), which Wikipedia is saying is invalid for non-nested distributions. The two test which seems to be a very similar test statistic (i.e. the difference in log-likelihoods, but the AIC introduces a parameter penalty) but assuming a different distribution for the significance of the difference. Why the difference in the test statistic's distribution?

As an additional question motivated by using the AIC vs L-ratio for comparing these two distributions, which one is correct for non-nested distributions and which is the better way to go in this case?

$\endgroup$
1

1 Answer 1

2
$\begingroup$

AIC IS NOT ONLY FOR NESTED MODELS. This is one of those fake news (myths) about AIC.

See: AIC Myths and Misunderstandings

There are several information criteria that can be used to compare different non nested models (BIC, DIC, ...).

You need to compare the models for the same data set. If you need to transform it for one of them, use a proper change of variable first, and then apply the model. For instance, for positive data, instead of using a normal on $\log(x_i)$, use a lognormal on $x_i$. This will take into account the always neglected Jacobian. This is: all models to be compared should be on the same scale and support.

If one model is not suitable for the data in the raw scale, it means that model is not good for the data, period.

$\endgroup$
3
  • $\begingroup$ Thank you for the ref to the AIC Myths and Understandings! Regarding the point on normal on $log(x_i)$ vs lognormal on $x_i$, I am fitting using the lnorm functions provided in poweRlaw which build on base R's dlnorm and plnorm functions, so this is correctly fitting the lognormal I guess! Both models are being compared over the same support $x_i \in [x_{min}, Inf)$. So then we are in good shape to directly compare the AIC? $\endgroup$
    – Heymans
    Aug 18, 2021 at 15:57
  • $\begingroup$ @samy Thanks for linking to the "AIC myths" summary. It says: "Several papers and books helped to isolate a few good methods (e.g., AIC, BIC, RIC) while indicating that other approaches were almost universally poor (e.g, $R^2$ stepwise hypothesis testing)." As stepwise variable testing (ANOVA) is presented as THE standard method in text books about linear regression, I were quite interested in these "several papers". Are you aware of surveys summarizing the results? $\endgroup$
    – cdalitz
    Aug 18, 2021 at 18:11
  • $\begingroup$ @cdalitz, the poor state of model selection sections in textbooks (namely, recommending stepwise methods or least presenting them uncritically) has been noticed in other threads as well. It should be possible to find if you are interested in that. $\endgroup$ Aug 19, 2021 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.