I looked around Cross Validated and I found pieces of information, but could not answer my particular issue.

I am working with data on detection of an animal species at different locations and seasons. Each "sampling unit" was sampled 3 times, so I am trying to fit my data using Occupancy Models (Darryl I. MacKenzie has extensive literature on the subject).

I have a set of covariates which I want to include in the model as possible explanatory variables for both detection and occupancy probabilities. I tried fitting all possible (and sensible) models, expecting to average them afterwards based on their AIC weights (following Burnham and Anderson). The problem is that they seem to be dramatically overfitted.

Then I found this publication [1] in which the authors recommend using a penalized likelihood approach to reduce overfitting, choosing the penalty parameter based on a k-fold cross validation. However, they do not mention the possibility of making multi-model inference.

Now I am trying to put the best of both worlds together, but I am not certain if this would be the right procedure:

  1. Pick a set of sensible values for then penalty parameter
  2. For each of these penalty values, try fitting all the candidate models on each of the k folds, calculating the average parameters, and evaluating the average model on the leave-out data.
  3. Look for the value of the penalty parameter for which the evaluation on the leave-out data got the highest likelihood.
  4. Take that value, fit all the candidate models using the whole dataset, and calculate the final averaged model.

Would that be the correct way to go? Are there any references worth reading about this?

[1] Hutchinson, R. A., Valente, J. J., Emerson, S. C., Betts, M. G., Dietterich, T. G. (2015), Penalized likelihood methods improve parameter estimates in occupancy models. Methods in Ecology and Evolution, 6: 949–959. doi: 10.1111/2041-210X.12368

  • $\begingroup$ How do you know that the models selected by AIC are overfitted? Also, did you use AIC or AICc? $\endgroup$ – James Sep 10 '15 at 19:20
  • $\begingroup$ @James AICc, actually. I have found two signs of overfitting: (1) without any penalization, max likelihood estimations yield ridiculously high values for the parameters; and (2) when I tried crossvalidation for determining the best penalty for the full model alone, it seemed that the best predictive model came from a strongly penalized model (i.e., with its parameters too much shrunk towards zero) $\endgroup$ – Marshall Sep 10 '15 at 19:26

I don't think it makes sense to do multimodel inference on top of penalized likelihood. If you start with the biggest model and adjust the penalty via CV, you will arrive at alternative to the averaged model of B&A.

In other words, when one relies on AIC, one tries to estimate the model's bias-variance tradeoff in sample, but with CV one tries to estimate it explicitly, out of sample. Combining these two doesn't make much sense: I would say that if it is possible to do CV at all, then that's what I'd prefer.

  • $\begingroup$ I assume from your answer that you are thinking of using a different penalization for each parameter. Otherwise, I cannot see they are equivalent.In the reference I found they apply a single penalization to the log-likelihood $-\lambda \sum_i \theta_i^2$, where $\theta_i$ are all the coefficients and intercepts in the model. Therefore, all parameters are equally shrunk. When averaging, less important parameters are further shrunk to zero than the important ones instead. $\endgroup$ – Marshall Sep 11 '15 at 21:18
  • $\begingroup$ I am not saying they are strictly equivalent, but the idea is similar. Note also that if you implement the unconditional variance estimator from Ch 4 of B&A, you will see that it generates a much higher variance than any individual model in the pool. That is, the models in the pool may appear overfitted and as a result each can generate a very small p-value, but the multimodel p-value will be much higher, and in that sense the multimodel inference will not be overfitted. $\endgroup$ – James Sep 14 '15 at 16:01
  • $\begingroup$ Sorry for pushing this on... The average model's estimators have greater variance indeed. But in this case that means the model can predict nothing, because the uncertainty in the parameters is huge. What I was after by using a penalized likelihood was to constrain the parameter space so that each model was fitted to an optimum within "sensible" values. $\endgroup$ – Marshall Sep 15 '15 at 21:53
  • $\begingroup$ What happens if you follow B&A's advice and restrict the model pool so that each model has at least 40 obs/parameter? $\endgroup$ – James Sep 16 '15 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.