Which weighting to use for regression analyses at different levels of aggregation

Question

I run a study with subjects in 400 groups of heterogeneous sizes ranging from 2 to 20 individuals. I have outcome data at the group level and at the individual level. Treatment was randomly assigned at the group level and my main interest is the average treatment effect on a group-level outcome. Groups are ``natural'' groups, i.e., unobservable might be correlated with group-size.

Problem: When I run individual-level regressions (to supplement my group-level analyses), subjects from larger groups mechanically make up a larger fraction of the sample, i.e., they are overrepresented in the estimation.

Idea: It seems appropriate to use weighted OLS to make sure that observations from each group weights the same total, and that results from the group-level regressions and the individual-level regressions are comparable. My idea is to use the inverse group-size as weights in the OLS, so that weights sum up to 1 for each group.

For those, used to using Stata. For the group-level data (~400 observations), I run

reg y_group treatment

and for the individual-level data (~400*10=4,000 observations):

bys group_id: inverse_groupsize = 1/_N
reg y_indiv treatment [pweight=inverse_groupsize], cluster(group_id)

Question:

1. Does this make sense conceptually or is there anything I need to worry about?
2. Are pweights the right way to do this in Stata?

P.S: I cross-posted that question on Statalist.org

Answer 1

You have a group-level treatment $X$ and an individual-level outcome $Y$ for $N$ individuals in $J$ groups, each with group size $n_j$ . When you regress a vector of length $J$ containing the group means $\bar{y}_j$ against another vector of length $J$ containing the corresponding treatments $x_j$, the information of each group is represented once (i.e. one group mean for each group). When you instead regress a vector of length $N$ containing the individual values $y_{ij}$ against another vector of length $N$ containing the corresponding $x_j$ values, the information from large groups has a large representation (i.e many individuals from each large group) and the information from small groups has a small representation (i.e. few individuals from each small group). This, in your opinion, makes the group-level analysis and the individual-level analysis "not comparable".

What you would like to have in the individual-level analysis is an equal representation of the groups. With $J=5$, you would like each group to represent $\frac{1}{5}$ of the cake. So if the first group has $n_1=10$, those ten individuals have to share $\frac{1}{5}$ of the cake, which means each individual gets a weight of $\frac{1}{5}/10=\frac{1}{50}$. In general, the weight you seem to be looking for is $\frac{1}{J\times n_j}$.

This seems like a bad idea because, by weighting individuals to make all the groups have equal relevance in your estimation, you assume that the information you can get from a group with $n_k=20$ is just as good as the information you can get from a group with $n_l=2$. Probability theory tells us that larger samples carry more information, so this assumption is very odd.

It occurs to me that, for the group-level analysis, you could instead create a vector of length $N$ containing the means $\bar{y}_j$ of each individual and regress it against another vector of length $N$ with the corresponding group treatments $x_j$. That way the groups would have the same representation as in your individual-level regression. But to be honest I have not seen that done before and I haven't worked very much with cluster-level treatments. You could also have a look at this paper and its "neighborhood effects" references.

(BONUS) If you really want to apply those $\frac{1}{J\times n_j}$ weights in Stata, then yes you can do it with pweight.

Answer 2

Have a look at the literature of cluster randomized trials, that seems to be the type of situations you face.

Essentially, uncertainty has (at least) two sources here: variability between groups and individuals within a group. By having fewer individuals in some groups, there is greater uncertainty about the mean outcome in that group. However, by having more individuals in a group, you can only reduce your uncertainty about the mean outcome in that group, but you still are not reducing the uncertainty due to the variability between groups - which you can do by having more groups. In fact, if you do not specifically care about the specific group you studied, but rather some generalized population of groups, then reducing the uncertainty about the mean response in each group is a less efficient (in terms of overall number of individuals needed) way of reducing uncertainty than having more groups (which does not have some lower limit on how much it can reduce variability).

If you are not adjusting for covariates at the individual level, then the weights you propose are essentially equivalent to doing a OLS of the individual patient data without adjusting for group. There would be no benefit in that and you would be ignoring between group variability. For that reason you could vastly overstate the evidence for a treatment effect due to between-group-variability.

One way to explicitly model the between group-variation in outcomes is a hierarchical model with between-group-variability. E.g. something like given $\mu_i$ we have for patient $j$ in group $i$ $$y_{ij} \sim N(\mu_i + \delta x_i, \sigma),$$ where $\delta$ is the treatment effect and $x_i$ is the indicator for whether group $i$ is on treatment or not, with the variation of the $\mu_i$ across groups given by $$\mu_i \sim N(\nu, \tau).$$