I have a number of different groups (on the order of 100). Each has many members (on the order of 1000) but the groups are all different sizes and range from 100 to 1000+ in size. Each of these groups is divided into a control set and a test set. The control set did not receive a treatment, and the test set received a treatment. Furthermore, the proportion of members that belong to the test and control sets varies for each group from 5% to 30% of the group size. I have a measure of interest, Y, that may or may not be affected by the treatment. However, Y is for sure affected by the group identity.
I want to now estimate the treatment effect, Y_test - Y_control for the whole population, combining all groups into one. If the control and test sets were all equal size I could just pool everyone into a single population to compute the treatment effect on the overall population. Because the groups are unequal size, because the proportion of members in the test set and control set is different for every group, and because the mean Y is also different for every group, I'm not sure how to compute this estimate.
One option is to just compute an estimate Y_test - Y_control for each group and then take the mean of that. That should return something reasonable with enough groups since Y_test - Y_control is not affected by the group identity. But I instead want to have access to the individual-level variance rather than only the group-level variance.
I have thought about a number of schemes but I'm not sure what is correct to do. For example I can do something like subtract the mean of the control set from each of the individual members in each segment, and then pool the individuals together into a single population. But this is problematic because it cannot distinguish between poor control set estimates and good ones. E.g. in an edge-case where a group has 1 control member and 1 million test members, I may get a treatment difference when it nevertheless is not real. Or in other words, it weighs a group with 100 control members and 1000 test members the same as a group with 1 control member and 1000 test members. And conversely, those two groups would be weighed equally if I instead removed the mean Y_test from each group. So this computation is not symmetric between choosing Y_test or Y_control as the center, and it should be.
How can I go about computing this?
So just to summarize the issues are that the groups are all different sizes, the control and test set splits within each group are also all different sizes, and the group identity itself modifies the measure of interest, mean Y, but it does NOT modify the difference Y_test - Y_control between test and control sets. And what I want to do is get the best estimate Y_test - Y_control that is representative of the whole population, and in such a way that I have access to the individual-level variance.