Can the AICc (corrected Akaike information criterion) measure be used for post-hoc analysis?

Question

I am collecting a lot of data from neurons and will be analyzing each neuron separately. Is it valid to pick the best of several models based on the AICc for each neuron and then use the ANOVA p-values from the best model? Or should should I only run the AICc on preliminary data and then use the best model analyze every neuron?

My impression was that it is only statistically valid to select the model in the preliminary experimental design phase, but I'm now having difficulty finding this explicitly stated. References would be greatly appreciated.

Answer 1

I believe I have found my own answer. It sounds like using AIC to select from a large set of conceivable variables can result in models that are "excessively" tailored to the data at hand, ie. data dredging. However, if selecting from a limited set of a priori models, then it should be fine.

Reference:

Question 4 from Joseph E. Cavanaugh's slides at: http://myweb.uiowa.edu/cavaaugh/ms_lec_14_ho.pdf

Answer 2

Here are a couple of good papers and books that discuss the use of AIC in the ecological disciplines. They emphasize the importance of a "little hard thinking" or "navel gazing" to develop an a priori model set.

Anderson, D. R., and K. P. Burnham. 2002. Avoiding pitfalls when using information-theoretic methods. Journal of Wildlife Management 66(3):912-918.

Anderson, D. R., K. P. Burnham, W. R. Gould, and S. Cherry. 2001. Concerns about finding effects that are actually spurious. Wildlife Society Bulletin 29(1):311-316.

Burnham, K. P., and D. R. Anderson. 2002. Model selection and multimodel inference: a practical information-theoretic approach. Second edition. Springer, New York, USA.

Anderson, D. R. 2008. Model based inference in the life sciences: a primer on evidence. Springer, New York, USA.