Say we want to test which of two products A and B is more liked by the general public, by obtaining ratings from random participants assigned to groups A and B, i.e. a between-subjects design. Let's assume that obtaining a rating for A renders the rating that someone would then give for B irrelevant, such that we could not do the experiment in a within-subject design.
How can we be sure that the difference in the average ratings of the two groups are really due to difference in the average likeability of the two products, and not merely to differences in the preferences of the people who happened to have been (randomly) assigned to the two respective groups?
Does the beteen-subjects ANOVA somehow account for this "risk"? As far as I understand, its sum-of-squares (SS) term accounts only for the variance due to the different conditions/manipulations (SS_between-subjects), whereas the variance due to subjects must remain as part of the error term. It would thus seem that the risk of wrongly attributing variance in the DV to the conditions themselves (when in fact it is due to the subjects differing at baseline) is not really accounted for.
It seems to me there would have to be some sort of "baseline" on which to ensure the two groups are matched, but unless extra measures are taken for that, do the inner workings of the ANOVA in any way provide such a baseline matching?
Asking the question differently, for the difference between the average ratings of A and B to be called "significant", do we in any way require this difference to be greater because of this "different people have different preferences" confound?