I have a number of different groups (on the order of 100). Each has many members (on the order of 1000) but the groups are all different sizes and range from 100 to 1000+ in size. Each of these groups is divided into a control set and a test set. The control set did not receive a treatment, and the test set received a treatment. Furthermore, the proportion of members that belong to the test and control sets varies for each group from 5% to 30% of the group size. I have a measure of interest, Y, that may or may not be affected by the treatment. However, Y is for sure affected by the group identity.

I want to now estimate the treatment effect, Y_test - Y_control for the whole population, combining all groups into one. If the control and test sets were all equal size I could just pool everyone into a single population to compute the treatment effect on the overall population. Because the groups are unequal size, because the proportion of members in the test set and control set is different for every group, and because the mean Y is also different for every group, I'm not sure how to compute this estimate.

One option is to just compute an estimate Y_test - Y_control for each group and then take the mean of that. That should return something reasonable with enough groups since Y_test - Y_control is not affected by the group identity. But I instead want to have access to the individual-level variance rather than only the group-level variance.

I have thought about a number of schemes but I'm not sure what is correct to do. For example I can do something like subtract the mean of the control set from each of the individual members in each segment, and then pool the individuals together into a single population. But this is problematic because it cannot distinguish between poor control set estimates and good ones. E.g. in an edge-case where a group has 1 control member and 1 million test members, I may get a treatment difference when it nevertheless is not real. Or in other words, it weighs a group with 100 control members and 1000 test members the same as a group with 1 control member and 1000 test members. And conversely, those two groups would be weighed equally if I instead removed the mean Y_test from each group. So this computation is not symmetric between choosing Y_test or Y_control as the center, and it should be.

How can I go about computing this?

So just to summarize the issues are that the groups are all different sizes, the control and test set splits within each group are also all different sizes, and the group identity itself modifies the measure of interest, mean Y, but it does NOT modify the difference Y_test - Y_control between test and control sets. And what I want to do is get the best estimate Y_test - Y_control that is representative of the whole population, and in such a way that I have access to the individual-level variance.

  • $\begingroup$ Try googling "inverse variance fixed effects meta-analysis" that may provide you the answer you want (assuming that treatments were assigned by randomization within each group etc.). It is not clear to me what you mean by "access to the individual-level variance rather than only the group-level variance" - do you mean that you want to assume that variances differ by group? Or something else? $\endgroup$ – Björn Oct 3 '16 at 7:05
  • $\begingroup$ I mean that one method to estimate Y_test - Y_control, since that difference doesn't depend on the group, is to take a mean of that difference over the groups. But if I do that, the appropriate statistical test is the variance available between the group estimates. So if there's 100 groups, I'd have 100 measures, and the variance of those measures would be used in statistical tests. What I want is to have the equivalent of just having a single giant group with a test set and a control set. That's a single difference, and instead the variance / stats use individual member measures. $\endgroup$ – CHP Oct 3 '16 at 7:14
  • $\begingroup$ So as I said, one way to accomplish that would be to center the mean of the control sets at 0 for each group, and subtract the control mean from each individual member's measurement within each group. Then pool all controls together and all tests together. This would then just be two groups being compared, with stats done on individual member measures and variance. Basically I want to do the analysis using individual member measures rather than the group level measures because the groups don't actually affect the treatment effect size. $\endgroup$ – CHP Oct 3 '16 at 7:16
  • $\begingroup$ So, why is a statistical model with group as a factor in the model (or for normally distributed data equivalently a fixed effects meta-analysis) not the right approach? That's still not clear to me. $\endgroup$ – Björn Oct 3 '16 at 8:47
  • $\begingroup$ Right, a model with treatment and group variables should work. But I was hoping I could come up with something simpler - e.g. a direct computation like those described. For example, if the control group was balanced across the entire population (50% control for each group) then I could a two group comparison. The reason I would prefer a simpler method is because the people who will use it would be more comfortable with that based on their work flow and what they're used to. But a valid answer to this question could be 'What you are asking for is not possible.' $\endgroup$ – CHP Oct 3 '16 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.