How can I compare my model to a technically invalid model?

Question

I've created nice little nonlinear model relating survival probability to length in salmon. I fit it assuming binomial errors and minimizing the negative log likelihood. I've been asked to compare it to someone else's model, where they binned the data and fit a straight line to it. However, the lowest bin includes the long left tail of the length distribution, and would predict 0 (or negative) chance of survival for those fish, were they not lumped into a bin with higher length average---but some of those fish do survive. That said, for some data sets, the linear model does quite well on the binned data.

I'd like to compare these models, but I can't use AIC because the linear model's invalidity makes its AIC explode. I could truncate the data--it is a very small proportion of the data, or I could bin the data and calculate an AIC for my model assuming normal errors, but I don't really feel great about either of those. Are there other options, or are these choices not so bad?

Answer 1

One solution would be to use cross-validation methods. This might be a conceptually easy (and elegant) solution because the model you have differs significantly from the model to be compared to. AIC or BIC won't really work here because the functional forms of these two models are very different -- yours is nonlinear and their model is not only linear but also based on binned data. AIC or BIC is insensitive to functional forms.

I wouldn't worry about binning vs non-binning too much, since it seems to me that binning is a modeling decision that could make a model better or worse. In other words, it's a feature whose effectiveness should be tested.

Now, assuming you can implement the other model, you can perform a k-fold cross-validation:

1. Divide your data into k subsets;
2. Iteratively leave one subset out, and train your model (without binning) and the other model (with binning) on the rest of the subsets;
3. Compute the sum of loglikelihoods of the subset that was left out in the previous with regard to your model and the other model. This should be relatively straight-forward: in your nonlinear model, error is binomially distributed; in the other model, error is normally distribution since it's a simple linear regression;
4. Repeat 2 and 3 until you have used each of the k subsets as the test subset (thus the name k-fold).
5. You can then compare which model gives you the better loglikelihood (i.e. the less negative one).