1
$\begingroup$

I've a county/district that I've divided into ~300 grids that are 15km^2 in size attributed with various habitat and economic variables that have been summarized and standardized. I then have 2 types of data 1) presence/absence of leopard caused human deaths and 2) counts of those human deaths. Of the 300 grids 80 had >=1 human death. I came up with 14 a-priori hypotheses to test.

Initially, I simply ran AIC model selection and averaged the best models. The reviewers said this was incomplete as I still need to validate the models. It was suggested I do k-fold cross validation (KCV) to validate the model and get at model performance or predictive accuracy. However, I am having difficulties figuring out how to do that when multiple models (say 3 models for example) are the "best" according to AIC and how you would do KCV over an averaged model in R. I've also been seeing that some people have used CV methods for model selection itself. That is, they select the "top model" based on the one that predicts the best instead of using AIC, BIC, etc. But, model selection using AIC and CV are asymptotically equivalent. That is, as sample size increase the AIC top model will converge to the "best predicting" top model selected using CV methods. How would I know if my sample size is "enough" to use CV instead of AIC?

Can I still use AIC - and then only run the KCV on the absolutely best top model? Or run the KCV over all three best models and then somehow compare between them?

I found the R package cvTools which has a cv.Select command, but I have yet to see any discussion of interpretation for the results or to know what it is exactly I am doing outside of the PDF that has the short short example.

Basically k-fold type techniques were developed for assessing predictive accuracy for a single model... I've got multiple models according to AIC that work -- so what to do?

Is the reviewer correct in saying I need to validate the model? If CV and AIC model selection are asymptotically equivalent. That is, as sample size increase the AIC top model will converge to the "best predicting" top model selected using CV methods -- would I even have to do this as CV would essentially give the same result?

$\endgroup$
1
$\begingroup$

If possible, you could randomly subset your data into a training and testing set. You could run your AIC on the training data, do your model averaging, and then compare the predictions to your testing set. Not exactly K-fold CV, but it would give you an idea of how well your model performs.

$\endgroup$
1
$\begingroup$

I'm clueless regarding your dataset or model building. But...

I think there might be some confusion of terminology here. I assume that the reviewer would like some form of internal validation - some measure of the optimism or over-fitting of the model. Just google "internal validation" or "steyerberg internal validation" and you should be set.

Random subset is a great recommendation since it is easy given your complex model building strategy. Unfortunately, unless you've got a huge dataset, it is not going to get you published. Bootstrapping is the gold standard. I assume the reviewer recommended k-fold assuming that the complex model building strategy employed would be difficult to recreate using bootstrap validation and that k-folds would be feasible alternative. But I don't really understand your model building - it may not be possible to replicate it in k-fold either. Then a subset would be very helpful and your only reasonable option I think.

Regarding AIC model selection: it is unclear why you don't think it is prone to over-fitting the data.

Hope there is something useful here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.