I looked around Cross Validated and I found pieces of information, but could not answer my particular issue.
I am working with data on detection of an animal species at different locations and seasons. Each "sampling unit" was sampled 3 times, so I am trying to fit my data using Occupancy Models (Darryl I. MacKenzie has extensive literature on the subject).
I have a set of covariates which I want to include in the model as possible explanatory variables for both detection and occupancy probabilities. I tried fitting all possible (and sensible) models, expecting to average them afterwards based on their AIC weights (following Burnham and Anderson). The problem is that they seem to be dramatically overfitted.
Then I found this publication  in which the authors recommend using a penalized likelihood approach to reduce overfitting, choosing the penalty parameter based on a k-fold cross validation. However, they do not mention the possibility of making multi-model inference.
Now I am trying to put the best of both worlds together, but I am not certain if this would be the right procedure:
- Pick a set of sensible values for then penalty parameter
- For each of these penalty values, try fitting all the candidate models on each of the k folds, calculating the average parameters, and evaluating the average model on the leave-out data.
- Look for the value of the penalty parameter for which the evaluation on the leave-out data got the highest likelihood.
- Take that value, fit all the candidate models using the whole dataset, and calculate the final averaged model.
Would that be the correct way to go? Are there any references worth reading about this?
 Hutchinson, R. A., Valente, J. J., Emerson, S. C., Betts, M. G., Dietterich, T. G. (2015), Penalized likelihood methods improve parameter estimates in occupancy models. Methods in Ecology and Evolution, 6: 949–959. doi: 10.1111/2041-210X.12368