1
$\begingroup$

I want to test the effects of different versions of treatment across multiple test groups of a given population. In order to do that, I want to randomly sample my population into, let's say, 4 groups of 2000+ subjects each.

After sampling, I generally try to evaluate each group for equivalence prior to assigning treatment and executing the test.

My question is on how to test for equivalence. In the past I have used ANOVA to evaluate differences of mean on a given attribute (for example, AGE or CHRONIC_CONDITION Y/N), or I have used t-tests to evaluate differences in attribute distribution...remember the goal here is to ensure that my groups are as "equivalent" as possible. However, I feel like there is probably a better way to measure similarity. ANOVA feels too squishy, and T-Tests become unmanageable with each version (for example, 4 groups would require 6 t-tests: AB, AC, AD, BC, BD, CD).

Are there better methods to evaluate sampled group equivalence or similarity? Bonus question: a way to do it in Python?

Note: I've searched the archive but haven't found exactly what I'm looking for. Happy to have another look if someone finds a thread.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

There are so-called equivalence tests. You will find related information on this site (), for example, How to test hypothesis of no group differences?.

However, note also that many believe that balance checking is senseless in randomized experiments (e.g., https://stats.stackexchange.com/a/199838/27276).

On a more practical side, your groups seem very large. Why would you expect any form of imbalance in truly randomized groups?

Last, might also be an option.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the reply. To answer your question, yes, there are occasions when samples are unbalanced. I am able to diagnose by evaluating a t-test against a particular metric (eg: age) and find the differences to be significant < 0.05. When I re-sample, the imbalance generally disappears. This may be a product of faulty randomization in my library - I'm using the sampler included with pandas, a python library. Without running the t-test, I would be at risk of moving forward with a test against imbalanced groups. $\endgroup$
    – Peter
    Feb 19, 2019 at 6:19
  • $\begingroup$ To rephrase stats.stackexchange.com/a/199838/27276 (see also the comments): In randomized assignments, we know by definition that groups are equal in the population. Thus, any significance test is senseless; if p < .05, it is a type I error. $\endgroup$
    – hplieninger
    Feb 19, 2019 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.