Comparing two regression model (Beta regression and linear regression)

I was informed that beta regression was more preferred to be applied to proportion data instead of linear regression.

I know I can use various ways to compare the goodness of fit of two models, such as lrtest, anova, AIC, BIC. However, I wonder if these methods can only be used in comparing models that are same type. (Beta regression vs Beta regression)

Can I compare the two following model with AIC or other method?

betareg(change ~ IV1 * IV2, data = dF)
lm(change ~ IV1 * IV2, data=dF)


Yes, Akaike Information Criterion (AIC) can be used. Comparing AIC values with different error distributions is not a problem.

Given we have a valid log-likelihood and the correct number of fitted parameters by our model, AIC is attainable. Nevertheless we should be careful about how the corresponding (log) likelihoods are calculated because often it is easier to omit normalising terms thus leading to correct convergence (when optimising Maximum Likelihood) but also to nonsensical values. Similarly, AIC is occasionally defined not by its standard definition of : $$-2 \log L + 2 k$$, where $$k$$ is the number of parameters and $$\log L$$ is the log-likelihood of the model but by employ some notion of the corresponding error variance $$\hat{\sigma}^2$$. This again perplexes things as commonly it is equivalent up to a constant (see this thread on "Different AIC definitions" for a detailed example). Finally, likelihood calculations are often conditional on initial states (ETS models being prime examples) or employ variants of ML (e.g. Restricted ML) that are not true likelihoods per se, so again something like this invalidates the direct comparison of model specific AIC values.

All in all, we can do such a comparison but we should be certain that we compare what we intended to. Computing a likelihood is not necessarily a trivial task and this can lead to problems.

Especially if the application of the model entails a prediction part, I would suggest using a resampling technique (e.g. bootstrap) or a simple hold-out dataset to get more insightful estimates of the model's performance.

A few final side-notes, in case they are forgotten:

1. We cannot compare AIC values between models that use difference response variables even if the one variable is a transformation of the other.
2. Whether or not, using AIC (or any other single number) to compare two models is a different question; usually choosing a "single number" approach is rarely the correct way.
3. I do not touch upon the issue of using different link functions. There is the question about "goodness-of-link" tests (see Pregibon, D. (1980) Goodness of link tests for generalized linear models for more details on this) and this can occasionally be influential especially if our sample is a bit abnormal (e.g. if we need to use Cauchy link function link = 'cauchyit' in betareg to account for data values approaching $$0$$ or $$1$$).
4. We are certain about the convergence of our estimator. If not, as it can be the case if we have a flat (log) likelihood surface, the values of the $$L$$ might be misleading and thus comparing AIC values is moot.
• Thank you for the answer. One question I would ask is about the second note. You mentioned about not using "single number approach", do you mean, I have to compare more than one number (Ex. comparing AIC, BIC.....)? – Oberon Jan 8 '19 at 5:12
• Cool. I am glad I could help. In many cases we might be tempted to us a single metric to choose a model. Or maybe two or $N$ metrics, etc. The point is that a model encapsulates our assumptions. We should not choose a model automatically but instead choose it in such that if facilitates our research question. AIC, $p$-values, etc. help us test or quantify certain assumptions but should not be our biggest factors on this. CV has some great threads on the perils of automatic model selection (eg. here). – usεr11852 Jan 8 '19 at 5:49
• Pleasure! If you find this answer helpful you could consider upvoting it or if it answers your question, accept it as an answer. If you need further clarifications you are welcome to ask. – usεr11852 Jan 10 '19 at 23:17

You can use BIC quite widely for any set of maximum likelihood-based models. However, in many applications, BIC simply reduces to maximum likelihood selection because the number of parameters is equal for the models of interest. This is the case in your example according to the code you write.

This post on Information Criteria is very informative. It suggest that BIC is basically a way of measuring out of sample prediction error without actually performing several modelfits.

Be that as it may, you could partition data and fit on one subsample while predicting for another and then calculate out of sample prediction error. One such approach is k fold cross validation.