2
$\begingroup$

Colleagues and I conducted an experiment with participants randomly allocated to each condition using survey software. Unfortunately, the cell sizes ended up being more unbalanced than we anticipated: Out of around 1743 participants, each randomized to one of 10 conditions (this was a 2x5 factorial design), 3 conditions ended up with just under 160 participants each, while two conditions wound up with 200 participants each. A quick simulation suggests only about a 6% chance of winding up with 3 or more conditions with fewer than 160 participants with this kind of randomisation regime, so we seem just to have gotten a bit unlucky.

I have seen advice to use type II or type III sums-of-squares for unbalanced ANOVA; we were planning to use type III anyhow as we are estimating an interaction (and checking that results are similar with type II if there is no interaction). Is this sufficient? Are there any rules-of-thumb we might follow, or analyses/simulations we might consider conducting, to assess whether this level of imbalance is likely to pose a problem?

Edit: Some additional context that may be helpful: For the analysis in question, we are testing the effect of four different kinds of information (and a no-information control), accompanying an either positive or negative COVID PCR test result, on participants' agreement with the proposition that the test recipient should self-isolate (on a 0 to 100 scale). We hypothesized an interaction with differences between positive/negative test results for some kinds of information but not others. And we got one, p < .001 - but wanting to ensure that this isn't somehow an artifact of our unbalanced numbers.

$\endgroup$
2
  • $\begingroup$ What hypothesis or hypotheses do you want to test? $\endgroup$
    – Dave
    Feb 26, 2021 at 20:32
  • $\begingroup$ Hi Dave - For the analysis in question, we are testing the effect of four different kinds of information (and a no-information control), accompanying an either positive or negative COVID PCR test result, on participants' agreement with the proposition that the test recipient should self-isolate (on a 0 to 100 scale). We hypothesized an interaction with differences between positive/negative test results for some kinds of information but not others. And we got one, p < .001 - but wanting to ensure that this isn't somehow an artifact of our unbalanced numbers. $\endgroup$
    – Gabriel
    Feb 26, 2021 at 20:45

1 Answer 1

1
$\begingroup$

To test the robustness of the analysis, I ended up repeating the analyses on a subset created by randomly sampling n from each cell, where n was the size of the smallest cell in the original experiment. Results were nearly identical.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.