Comparing two regression model (Beta regression and linear regression)

Question

I was informed that beta regression was more preferred to be applied to proportion data instead of linear regression.

I know I can use various ways to compare the goodness of fit of two models, such as lrtest, anova, AIC, BIC. However, I wonder if these methods can only be used in comparing models that are same type. (Beta regression vs Beta regression)

Can I compare the two following model with AIC or other method?

betareg(change ~ IV1 * IV2, data = dF)
lm(change ~ IV1 * IV2, data=dF)

Any answer will be appreciated.

Answer 1

Yes, Akaike Information Criterion (AIC) can be used. Comparing AIC values with different error distributions is not a problem.

Given we have a valid log-likelihood and the correct number of fitted parameters by our model, AIC is attainable. Nevertheless we should be careful about how the corresponding (log) likelihoods are calculated because often it is easier to omit normalising terms thus leading to correct convergence (when optimising Maximum Likelihood) but also to nonsensical values. Similarly, AIC is occasionally defined not by its standard definition of : $-2 \log L + 2 k$, where $k$ is the number of parameters and $\log L$ is the log-likelihood of the model but by employ some notion of the corresponding error variance $\hat{\sigma}^2$. This again perplexes things as commonly it is equivalent up to a constant (see this thread on "Different AIC definitions" for a detailed example). Finally, likelihood calculations are often conditional on initial states (ETS models being prime examples) or employ variants of ML (e.g. Restricted ML) that are not true likelihoods per se, so again something like this invalidates the direct comparison of model specific AIC values.

All in all, we can do such a comparison but we should be certain that we compare what we intended to. Computing a likelihood is not necessarily a trivial task and this can lead to problems.

Especially if the application of the model entails a prediction part, I would suggest using a resampling technique (e.g. bootstrap) or a simple hold-out dataset to get more insightful estimates of the model's performance.

A few final side-notes, in case they are forgotten:

1. We cannot compare AIC values between models that use difference response variables even if the one variable is a transformation of the other.
2. Whether or not, using AIC (or any other single number) to compare two models is a different question; usually choosing a "single number" approach is rarely the correct way.
3. I do not touch upon the issue of using different link functions. There is the question about "goodness-of-link" tests (see Pregibon, D. (1980) Goodness of link tests for generalized linear models for more details on this) and this can occasionally be influential especially if our sample is a bit abnormal (e.g. if we need to use Cauchy link function link = 'cauchyit' in betareg to account for data values approaching $0$ or $1$).
4. We are certain about the convergence of our estimator. If not, as it can be the case if we have a flat (log) likelihood surface, the values of the $L$ might be misleading and thus comparing AIC values is moot.

Answer 2

You can use BIC quite widely for any set of maximum likelihood-based models. However, in many applications, BIC simply reduces to maximum likelihood selection because the number of parameters is equal for the models of interest. This is the case in your example according to the code you write.

This post on Information Criteria is very informative. It suggest that BIC is basically a way of measuring out of sample prediction error without actually performing several modelfits.

Be that as it may, you could partition data and fit on one subsample while predicting for another and then calculate out of sample prediction error. One such approach is k fold cross validation.