Is it possible to compare model fit for a gaussian vs binomial glm

Question

I'm modelling some proportional data using a binomial GLM - ie fitted values are sigmoidal - but the data look like they may be better fit by a straight line (model validation plots of a simple linear model suggest this is ok too).

Is there any way to compare these models to see if one fits better than the other? I'm using r, and was originally thinking I could use

m1 <- glm(y ~ x, family = gaussian)

m2 <- glm(y ~ x, family = binomial)

then use AIC to compare them. (Note that covariates are the same for m1 & m2).

However, I've realised that these 2 models require the response variable to be structured differently - as a proportion in m1, and a matrix containing a 'success' and a 'failure' column for m2. I think (?) this means that this approach is inappropriate.

Is there a way I can test whether one model fits the data better than the other?

Thank you for your help!

Jay

I should also note - as far as my understanding of the relationships go, there isn't a theoretical reason to choose a sigmoidal relationship over a simple linear one, beyond the fact binomial errors are appropriate for proportional data.

Answer 1

Well first things first, you can try eyeballing the residuals and see if one or other model makes them more even, but I suspect given that you haven't seen a clear winner, that won't split them either.

In which case I think this really comes down to your definition of "better". Because you are using the same explanatory variables for both models, and the same number of parameters etc. then it is simply a matter of comparing the goodness of fit of the two models. You simply have to define what you mean by a good fit.

For example, if your utility is to minimise the mean square error of predictions, then pick the model that does that best - which should be the gaussian. If on the otherhand you want the one that minimses the mean square log odds error, then pick the one that does that best (this should be the binomial). Of course, since you are choosing a model for empirical pragmatic reasons, you needn't be restricted to this. You could choose the model that minimises the chi-squared error, or something even more exotic.

What are you going to use the model for when it is fitted? Really you want to think about what the cost to the user of the model is, when the model is wrong.

This is related in spirit to this question about cost functions: Cost function for validating Poisson regression models