Say I have a series of different regression models (e.g. 8 different models) which I conduct using data collected at a series of multiple locations (same predictor same response variables) and I have resultant AIC values from all of the predictors. I know which of the models has the lowest AIC. Is it logical to compare proportions of observed with lowest AIC to an expected value of evenly distributed proportions across all models? You could use a Z-test to test this. Or am I missing something? This might tell you if a model might have universal applicability or not IF only one of the models was significantly different from the expected value.

  • $\begingroup$ My understanding is that AIC is only useful when comparing different models for the same data set. If you compare models using results from different data sets, such as data taken at different locations, then AIC is not a useful tool for model selection. Can one of the statisticians please weigh in on this? $\endgroup$ – James Phillips Mar 22 at 14:53
  • $\begingroup$ Do I understand you correctly: say you have 100 data sets, you fit each with 8 models, and for each data set record which model is the best based on the AIC? Then you can of course say, e.g., "in 40% of the data sets, model_1 was selected" etc. So you have eight proportions, e.g., [0.4, 0.1, 0.1, ...] and want to check if they can come from a uniform distribution, right? $\endgroup$ – corey979 Mar 22 at 15:11
  • $\begingroup$ There are 100 data sets each using the a series of similar for example 8 models. Each of the 8 models across all data sets has the same response variable and the same set of predictors. So essentially I am looking for common models across the different data sets rather than a particular model for a specific data set. I am not comparing AICs between data sets directly but rather the observed frequencies of models selected on the basis of lowest AIC. $\endgroup$ – mlane Mar 22 at 16:09
  • $\begingroup$ Essentially what you said is correct $\endgroup$ – mlane Mar 22 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.