3
$\begingroup$

I was informed that beta regression was more preferred to be applied to proportion data instead of linear regression.

I know I can use various ways to compare the goodness of fit of two models, such as lrtest, anova, AIC, BIC. However, I wonder if these methods can only be used in comparing models that are same type. (Beta regression vs Beta regression)

Can I compare the two following model with AIC or other method?

betareg(change ~ IV1 * IV2, data = dF)
lm(change ~ IV1 * IV2, data=dF)

Any answer will be appreciated.

$\endgroup$
0

2 Answers 2

2
$\begingroup$

Yes, Akaike Information Criterion (AIC) can be used. Comparing AIC values with different error distributions is not a problem.

Given we have a valid log-likelihood and the correct number of fitted parameters by our model, AIC is attainable. Nevertheless we should be careful about how the corresponding (log) likelihoods are calculated because often it is easier to omit normalising terms thus leading to correct convergence (when optimising Maximum Likelihood) but also to nonsensical values. Similarly, AIC is occasionally defined not by its standard definition of : $-2 \log L + 2 k$, where $k$ is the number of parameters and $\log L$ is the log-likelihood of the model but by employ some notion of the corresponding error variance $\hat{\sigma}^2$. This again perplexes things as commonly it is equivalent up to a constant (see this thread on "Different AIC definitions" for a detailed example). Finally, likelihood calculations are often conditional on initial states (ETS models being prime examples) or employ variants of ML (e.g. Restricted ML) that are not true likelihoods per se, so again something like this invalidates the direct comparison of model specific AIC values.

All in all, we can do such a comparison but we should be certain that we compare what we intended to. Computing a likelihood is not necessarily a trivial task and this can lead to problems.

Especially if the application of the model entails a prediction part, I would suggest using a resampling technique (e.g. bootstrap) or a simple hold-out dataset to get more insightful estimates of the model's performance.

A few final side-notes, in case they are forgotten:

  1. We cannot compare AIC values between models that use difference response variables even if the one variable is a transformation of the other.
  2. Whether or not, using AIC (or any other single number) to compare two models is a different question; usually choosing a "single number" approach is rarely the correct way.
  3. I do not touch upon the issue of using different link functions. There is the question about "goodness-of-link" tests (see Pregibon, D. (1980) Goodness of link tests for generalized linear models for more details on this) and this can occasionally be influential especially if our sample is a bit abnormal (e.g. if we need to use Cauchy link function link = 'cauchyit' in betareg to account for data values approaching $0$ or $1$).
  4. We are certain about the convergence of our estimator. If not, as it can be the case if we have a flat (log) likelihood surface, the values of the $L$ might be misleading and thus comparing AIC values is moot.
$\endgroup$
5
  • $\begingroup$ Thank you for the answer. One question I would ask is about the second note. You mentioned about not using "single number approach", do you mean, I have to compare more than one number (Ex. comparing AIC, BIC.....)? $\endgroup$
    – Oberon
    Jan 8, 2019 at 5:12
  • $\begingroup$ Cool. I am glad I could help. In many cases we might be tempted to us a single metric to choose a model. Or maybe two or $N$ metrics, etc. The point is that a model encapsulates our assumptions. We should not choose a model automatically but instead choose it in such that if facilitates our research question. AIC, $p$-values, etc. help us test or quantify certain assumptions but should not be our biggest factors on this. CV has some great threads on the perils of automatic model selection (eg. here). $\endgroup$
    – usεr11852
    Jan 8, 2019 at 5:49
  • $\begingroup$ Pleasure! If you find this answer helpful you could consider upvoting it or if it answers your question, accept it as an answer. If you need further clarifications you are welcome to ask. $\endgroup$
    – usεr11852
    Jan 10, 2019 at 23:17
  • $\begingroup$ @usεr11852 Why AIC can be used in this case? The linear model assumes unbounded responses and beta regression model assumes bounded response in (0,1). The distributions do not even live with the same support. Had AIC been approximation to KL divergence, under this situation KL would be infinite. $\endgroup$
    – user45765
    Apr 4 at 1:45
  • $\begingroup$ (I still remember that when I first tried to answer this answer I had the same intuition as you) In the AIC calculation we care to have valid (negative) log-likelihood for each model. Both (maximised) likelihood values we attain are equally valid, thus comparable, despite not sharing the same support. Finally, particular to your last point: A KL divergence is between our model and a "real" model, not between two different candidate models. And the only information about a real model comes from the observations themselves, which are common for both candidate models. $\endgroup$
    – usεr11852
    Apr 4 at 2:57
0
$\begingroup$

You can use BIC quite widely for any set of maximum likelihood-based models. However, in many applications, BIC simply reduces to maximum likelihood selection because the number of parameters is equal for the models of interest. This is the case in your example according to the code you write.

This post on Information Criteria is very informative. It suggest that BIC is basically a way of measuring out of sample prediction error without actually performing several modelfits.

Be that as it may, you could partition data and fit on one subsample while predicting for another and then calculate out of sample prediction error. One such approach is k fold cross validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.