Suppose you estimate a model using firm-level or state-level data and then apply the estimates at a lower level of aggregation, say at factory-level$^*$ or county-level. If it makes things easier, imagine this is a model describing output of widgets Y given some number of inputs (X and Z).

I would like to know:

  1. Is there a name for this?
  2. Is this always a bad idea?
  3. What if it is not a lower level of aggregation, but merely a different level of aggregation (say model US state data, but use the model on CBSA data, ignoring the fact that not all of the US is in some CBSA)?

I think this is related to external validity and the ecological fallacy, but perhaps there is something more specific.

$^*$Assuming each firm has some number of factories.

  • $\begingroup$ The short answer, expounded in my answer below with extracts and sources (with links also, albeit largely to the econometric literature) is Aggregate Analysis. There are also interesting parallels in viewing the aggregation problem between the econometric literature (and other fields) and spatial statistics (per the source alluded to in Thomas's answer). $\endgroup$ – AJKOER Nov 25 '20 at 11:37

The assumption that the relationships are the same at a finer level of aggregation is exactly the ecological fallacy. The problem, more generally, of the relationship depending on how you aggregate is the Modifiable Areal Unit Problem

  • $\begingroup$ Amazingly, related to my answer below, your provided source notes; "MAUP can be used as an analytical tool to help understand spatial heterogeneity and spatial autocorrelation." akin to my answer, which covers in econometrics the use of the term "micro-level heterogeneity" also quoting your source "This topic is of particular importance because in some cases data aggregation can obscure a strong correlation between variables, making the relationship appear weak or even negative" highlights an embarrassing result in the econometric literature link to in my answer below as "conflicting results". $\endgroup$ – AJKOER Nov 25 '20 at 11:21

+1 to Thomas' answer.

That said, this is not always a bad idea. For instance, in forecasting, we frequently have a large number of noisy time series that we can reasonably expect to share some common dynamics. In such cases, it's common practice to estimate these common dynamics on an aggregate level and then impose them on the separate series we are interested in.

A common example is the impact of yearly seasonality on retail sales: you see the seasonality on, say, ice cream well enough if you aggregate over multiple stock keeping units (SKUs) and stores, but often not on the disaggregate SKU $\times$ store level. So people will aggregate total sales, estimate seasonality and push this down to the disaggregate series. This approach typically helps forecasting accuracy.

In the end, this is again a case of the bias-variance tradeoff: this idea will inject some bias into the lower level models, but reduce variance, compared to estimating (say) seasonality on the lower levels. But then again, not including seasonality on the lower levels will do exactly the same. Either approach may be better than modeling seasonality on the disaggregate level - or they may both be worse, depending on the situation.

  • 1
    $\begingroup$ I will just add to this that population comparisons have been very useful in dietary epidemiology because population-average diet is much more accurately measurable than individual diet and this compensates to a substantial degree for the potential aggregation bias. Prentice and Sheppard write about it here: academic.oup.com/biomet/article-abstract/82/1/113/254942 $\endgroup$ – Thomas Lumley Nov 25 '20 at 20:23

The question is addressed generally in the field referred to Aggregate Analysis. Here, for example, is an extract from a paper in this area:

Aggregate analysis has been established as a standard method on the study of market response behavior for a long time. Aggregation has advanced our understanding of the linkages among social characteristics and aggregate response behavior. However, aggregate analysis has been hindered by fragmentary and unsystematic procedures to determine the most appropriate level of aggregation. The general objective of this paper is to provide a conceptual framework to determine the level of aggregation of variables in data analysis. In addition, statistical procedures are suggested in this framework to verify and to determine the level of aggregation represented by a variable. The conceptual framework is useful for deciding if the variables are to be analyzed from micro-analysis focus or macro-analysis focus. The statistical procedures enable the researcher to systematically identify and verify the level(s) of aggregation of variables in an existing data set.

A key take-away is whether "variables are to be analyzed from micro-analysis focus or macro-analysis focus".

My personal experience upon applying a macro company developed prediction model at the field office level, and then aggregating for a hopefully a better company-wide forecast, proved to be somewhat unsuccessful. There can be apparently different cross-currents occurring locally (perhaps requiring an expanded model). In the literature, there is also a reference to micro-level heterogeneity, which upon aggregation may (or may not) largely cancel. With luck, one can achieve a parsimonious model that is actually more accurate forecasting with company-level data. It may also avoid producing conflicting results. Generally speaking, model misspecification occurring at the local level may result in bias, which upon aggregating, could degrade forecast quality.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.