0
$\begingroup$

I have mostly worked with Observational data where the treatment assignment was not randomized. In the past, I have used PSM, IPTW to balance and then calculate ATE. My problem is: Now I am working on a problem where the treatment assignment is randomized meaning there won't be a confounding effect. But treatment and control groups have different sizes. There's a bucket imbalance.

Now should I just analyze the data as it is and run statistical significance and Statistical power test? Or shall I balance the imbalance of sizes between the treatment and control using let's say covariate matching and then run significance tests?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Having different sample sizes in each group won't interfere with causal inference arguments. The statistical power under the alternative will be highest when comparing two groups with equal sample size. The p-value and the type I error rate under the null are not affected by imbalanced sample size when using a valid test.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks, Geoffery. Type II error and statistical power will be affected by the imbalance. Assuming the treatment group is ~15-20% of the total control group, in which case the statistical power will be lower? Is balancing the data to 50-50% before analysis is incorrect? $\endgroup$
    – manish Prasad
    Dec 8, 2021 at 19:19
  • 1
    $\begingroup$ Yes to both questions. Typically with IPTW we are balancing the distribution of patient features between treatment groups. We could create the weights so that they sum to any $n$ of choice in each group. However, this is artificially augmenting the sample size, imagining the sample size was something different from what was actually collected. When implementing IPTWs I recommend constructing the weights so they sum to the actual sample size in each group. Therefore, in your setting that does not require balancing of patient features there would be no IPTW adjustment. $\endgroup$
    – Geoffrey Johnson
    Dec 8, 2021 at 19:26
  • $\begingroup$ Thanks a lot, Geoffery, really helpful. Can the high confidence interval level with the imbalance and low power be considered statistically significant? $\endgroup$
    – manish Prasad
    Dec 9, 2021 at 9:29
  • 1
    $\begingroup$ Statistical significance refers to the p-value threshold or type I error rate of the test under the null hypothesis for decision making. You can simply report the p-value and confidence interval without declaring a "reject" or "not reject" decision about a particular hypothesis. You could instead make a "reject" or "not reject" decision about a particular hypothesis based on a particular p-value threshold. $\endgroup$
    – Geoffrey Johnson
    Dec 9, 2021 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.