Problems due to analyzing variables from different levels at one single level

Question

Please ease the following paragraph from the first chapter , Introduction to Multilevel Analysis , p.3 of the book:

Historically , multilevel problems have led to analysis approaches that moved all variables by aggregation or disaggregation to one single level of interest followed by an ordinary multiple regression , analysis of variance , or some other 'standard' analysis method . However , analyzing variables from different levels at one single common level is inadequate , and leads to two distinct types of problems .

The first problem is statistical .

If data are aggregated , the result is that different data values from many sub-units are combined into fewer values for fewer higher-level units . As a result , much information is lost, and the statistical analysis loses power .

On the other hand , if data are disaggregated , the result is that a few data values from a small number of super-units are 'blown up' into many more values for a much larger number of sub-units . Ordinary statistical tests treat all these disaggregated data values as independent information from the much larger sample of sub-units . The proper sample size for these variables is of course the number of higher level units . Using the larger number of disaggregated cases for the sample size leads to significance tests that reject the null-hypothsis far more often than the nominal alpha level suggests . In other words : investigators come up with many 'significant' results that are totally spurious .

EDIT : My question is ,

(1) In aggregation , how is information lost ? And how do we loose statistical power ?

(2) In disaggregation , how does the issue of sample size occur ? Do they mean if we use the sample size of lower level when we do actually need to use the sample size of higher level , then the sample size becomes large which increases the chance of rejecting the true null hypothesis ?

Answer 1

Here is a paper which demonstrates what can happen when all variables are analyzed on the same level. One can end up with the wrong conclusions. The simplest way to investigate this s to simulate some simple data, where variables on different levels have effects on a response variable in different directions. Red State Blue State is a book-length example of this importance.

Below is some simple R code to simulate simple data giving an example. You can just play with code, which should be self-explaining.

expit <- function(x) 1/(1+exp(-x))  
gen_mix_data <- function(n_groups=100, group_size=50, alpha=-0.8, beta=0.3, constant=-0.3) {
    gamma <- rep(rnorm(n_groups), rep(group_size, n_groups)) # random effects
    px <- expit(alpha*gamma)
    x <- rbinom(n_groups*group_size, 1, px)
    py <- expit(constant+gamma + beta*x)
    y <- rbinom(n_groups*group_size, 1, py)
    data.frame(x=x, y=y, group=as.factor(rep(1:n_groups, rep(group_size, n_groups))))
}

set.seed(7*11*13)
simdata <- gen_mix_data()

library(lme4)  

mod1 <- glmer(y ~ x + (1 | group), data=simdata, family=binomial())

mod0 <- glm(y ~ x, data=simdata, family=binomial)

Answer 2

1.

One example of loss of information when we aggregate is Simpson's paradox, in which we can see a different result altogether.

2.

Imagine breaking up a group into 20 partitions randomly; maybe one of them might randomly turn up to be significant.

As an extreme case if you break up your points which happen to be leverage points, this can lead to interesting behaviour.