Cross validation for determining likelihood penalization in multi-model inference

I looked around Cross Validated and I found pieces of information, but could not answer my particular issue.

I am working with data on detection of an animal species at different locations and seasons. Each "sampling unit" was sampled 3 times, so I am trying to fit my data using Occupancy Models (Darryl I. MacKenzie has extensive literature on the subject).

I have a set of covariates which I want to include in the model as possible explanatory variables for both detection and occupancy probabilities. I tried fitting all possible (and sensible) models, expecting to average them afterwards based on their AIC weights (following Burnham and Anderson). The problem is that they seem to be dramatically overfitted.

Then I found this publication [1] in which the authors recommend using a penalized likelihood approach to reduce overfitting, choosing the penalty parameter based on a k-fold cross validation. However, they do not mention the possibility of making multi-model inference.

Now I am trying to put the best of both worlds together, but I am not certain if this would be the right procedure:

1. Pick a set of sensible values for then penalty parameter
2. For each of these penalty values, try fitting all the candidate models on each of the k folds, calculating the average parameters, and evaluating the average model on the leave-out data.
3. Look for the value of the penalty parameter for which the evaluation on the leave-out data got the highest likelihood.
4. Take that value, fit all the candidate models using the whole dataset, and calculate the final averaged model.

Would that be the correct way to go? Are there any references worth reading about this?

[1] Hutchinson, R. A., Valente, J. J., Emerson, S. C., Betts, M. G., Dietterich, T. G. (2015), Penalized likelihood methods improve parameter estimates in occupancy models. Methods in Ecology and Evolution, 6: 949–959. doi: 10.1111/2041-210X.12368

• How do you know that the models selected by AIC are overfitted? Also, did you use AIC or AICc? Sep 10 '15 at 19:20
• @James AICc, actually. I have found two signs of overfitting: (1) without any penalization, max likelihood estimations yield ridiculously high values for the parameters; and (2) when I tried crossvalidation for determining the best penalty for the full model alone, it seemed that the best predictive model came from a strongly penalized model (i.e., with its parameters too much shrunk towards zero) Sep 10 '15 at 19:26

• I assume from your answer that you are thinking of using a different penalization for each parameter. Otherwise, I cannot see they are equivalent.In the reference I found they apply a single penalization to the log-likelihood $-\lambda \sum_i \theta_i^2$, where $\theta_i$ are all the coefficients and intercepts in the model. Therefore, all parameters are equally shrunk. When averaging, less important parameters are further shrunk to zero than the important ones instead. Sep 11 '15 at 21:18