0
$\begingroup$

Say we want to test which of two products A and B is more liked by the general public, by obtaining ratings from random participants assigned to groups A and B, i.e. a between-subjects design. Let's assume that obtaining a rating for A renders the rating that someone would then give for B irrelevant, such that we could not do the experiment in a within-subject design.

How can we be sure that the difference in the average ratings of the two groups are really due to difference in the average likeability of the two products, and not merely to differences in the preferences of the people who happened to have been (randomly) assigned to the two respective groups?

Does the beteen-subjects ANOVA somehow account for this "risk"? As far as I understand, its sum-of-squares (SS) term accounts only for the variance due to the different conditions/manipulations (SS_between-subjects), whereas the variance due to subjects must remain as part of the error term. It would thus seem that the risk of wrongly attributing variance in the DV to the conditions themselves (when in fact it is due to the subjects differing at baseline) is not really accounted for.

It seems to me there would have to be some sort of "baseline" on which to ensure the two groups are matched, but unless extra measures are taken for that, do the inner workings of the ANOVA in any way provide such a baseline matching?

Asking the question differently, for the difference between the average ratings of A and B to be called "significant", do we in any way require this difference to be greater because of this "different people have different preferences" confound?

$\endgroup$
1
$\begingroup$

How can we be sure that the difference in the average ratings of the two groups are really due to difference in the average likeability of the two products, and not merely to differences in the preferences of the people who happened to have been (randomly) assigned to the two respective groups?

At the population level, this is, in fact, impossible. That's the value of random assignment. When subjects are randomly assigned to conditions, then the conditions must be drawing on the same population, because the assignment to conditions is guaranteed to be independent of any features of the subjects. Any population difference in outcomes must be causal effects of the conditions themselves, and nothing prior to that.

It's possible that you happen to be unlucky such that, in your sample (as opposed to the population), the conditions are highly imbalanced with respect to some pre-intervention characteristic. But this is a kind of sampling error that standard statistical methods, like hypothesis tests, are supposed to help you with.

$\endgroup$
8
  • $\begingroup$ Thanks! So how specifically does the between-subjects ANOVA help with the sampling error that might lead to (at the sample level) a difference in the DV not being the causal effect of the conditions themselves? $\endgroup$ – z8080 May 4 '17 at 10:30
  • 1
    $\begingroup$ @z8080 The smaller the sample, the more likely it is that an observed between-conditions effect is due to bad luck in condition assignment. And so the smaller the sample, the larger an effect that a hypothesis test (like the $F$-test of an ANOVA) requires to declare significance. $\endgroup$ – Kodiologist May 4 '17 at 15:31
  • $\begingroup$ I recognise the principles you state from past stats courses, but I guess I am just confused as to why it is they pertain to the above. Is it fair to say that the answer to my question is simply: as long as the allocation to groups is done truly randomly, and as long as the sample size per group is large enough (given expected effect size & power), then you can really trust a group difference to be causally due to the conditions themselves having different liking in the population, as opposed to merely things being confounded by the groups happening to have different preferences for them? $\endgroup$ – z8080 May 4 '17 at 19:17
  • 1
    $\begingroup$ @z8080 Yeah, that's a pretty fair statement. $\endgroup$ – Kodiologist May 4 '17 at 20:26
  • 1
    $\begingroup$ @z8080 The details depend on the specific data-analytic method, but generally the idea is that you look at differences within subjects, which will naturally remove any per-subject additive noise term that's the same at both measurements. $\endgroup$ – Kodiologist May 15 '17 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.