I've a county/district that I've divided into ~300 grids that are 15km^2 in size attributed with various habitat and economic variables that have been summarized and standardized. I then have 2 types of data 1) presence/absence of leopard caused human deaths and 2) counts of those human deaths. Of the 300 grids 80 had >=1 human death. I came up with 14 a-priori hypotheses to test.
Initially, I simply ran AIC model selection and averaged the best models. The reviewers said this was incomplete as I still need to validate the models. It was suggested I do k-fold cross validation (KCV) to validate the model and get at model performance or predictive accuracy. However, I am having difficulties figuring out how to do that when multiple models (say 3 models for example) are the "best" according to AIC and how you would do KCV over an averaged model in R. I've also been seeing that some people have used CV methods for model selection itself. That is, they select the "top model" based on the one that predicts the best instead of using AIC, BIC, etc. But, model selection using AIC and CV are asymptotically equivalent. That is, as sample size increase the AIC top model will converge to the "best predicting" top model selected using CV methods. How would I know if my sample size is "enough" to use CV instead of AIC?
Can I still use AIC - and then only run the KCV on the absolutely best top model? Or run the KCV over all three best models and then somehow compare between them?
I found the R package cvTools which has a cv.Select command, but I have yet to see any discussion of interpretation for the results or to know what it is exactly I am doing outside of the PDF that has the short short example.
Basically k-fold type techniques were developed for assessing predictive accuracy for a single model... I've got multiple models according to AIC that work -- so what to do?
Is the reviewer correct in saying I need to validate the model? If CV and AIC model selection are asymptotically equivalent. That is, as sample size increase the AIC top model will converge to the "best predicting" top model selected using CV methods -- would I even have to do this as CV would essentially give the same result?