# Is it legitimate to use AIC to compare models on the same data at different levels of aggregation?

I am doing some work on aggregation bias and MAUP. I have ols linear regressions on the same spatial data at 20 or so levels of aggregation-by-neighborhood-mean, from very fine to very coarse. I have been looking at all regression stats across those levels, but I am interested in whether the AICs are truly comparable, since the data isn't exactly the same for each model (same base data, different groupings).

('AIC ', -376690.80011657765) (original scale)
('AIC ', -10072.321299273639)
('AIC ', -2885.1160897361483)
('AIC ', -1329.8105840994881)
('AIC ', -708.88016385281207)
('AIC ', -492.45296306496391)
('AIC ', -308.68371624072711)
('AIC ', -218.0536153540844)
('AIC ', -166.20952785172358)
('AIC ', -118.37257148698347) (very coarse)


Your question is similar to this one. In general, any of the information criteria are only comparable for models using the exact same data, as they are functions of the likelihood, and the likelihood is a direct function of the data elements. If you were comparing models with different aggregations of parameters, like hierarchical models or using different distributional families but applied to the same data points, then the XIC criteria are a valuable comparison metric.

However, in your case, you are changing the underlying data used when fitting the model. You have a different number of data points when you look at your data on a finer or coarser level, thus the huge discrepancy in likelihood. Therefore, even using the same distributional family, your models are not comparable in the XIC framework. This is not the same as nested parameters on the same data, this is different data.

Remember, all of the XIC criteria are intended to find models which minimize the Kullback–Leibler divergence of said model of the underlying probability distribution from the "true" underlying probability distribution (in the case of AIC and its descendants) or from the "best" candidate model of the underlying probability distribution(in the case of BIC and its descendants). If you are dealing with two different data sets, how can you compare the underlying probability distribution of one data set with another? Only on the same data can we reasonably talk about finding the "best" candidate for the underlying probability distribution of that data.

• Ah, thanks so much. I had printed out the likelihoods, and they collapse similarly, but I hadn't understood that they were summed, as it says in the answer to the linked question, so of course the disaggregate ones are much higher. – J Kelly Dec 28 '16 at 16:38

We can't compare AIC for models based on different data. Memorize that $AIC=2k-2ln(L)$ so it depends on value of log-likelihood function. It's value and structure depends on dataset applied.

Therefore, you may compare AIC correctly only for models that use absolutely identical data, in order to investigate which set of parameters seems to be the most reasonable.

Also pay attention that difference in AIC among your models is very huge that is not typical. So the difference between original scale and very coarse scale data seems to be very big.