# How should one interpret the comparison of means from different sample sizes?

Take the case of book ratings on a website. Book A is rated by 10,000 people with an average rating of 4.25 and the variance $\sigma = 0.5$. Similarly Book B is rated by 100 people and has a rating of 4.5 with $\sigma = 0.25$.

Now because of the large sample size of Book A the 'mean stabilized' to 4.25. Now for 100 people, it may be that if more people read Book B the mean rating may fall to 4 or 4.25.

• how should one interpret the comparison of means from different samples and what are the best conclusions one can/should draw?

For example - can we really say Book B is better than Book A.

-
Are you specifically interested in the rating context? –  Jeromy Anglim Jun 29 '12 at 5:21
@JeromyAnglim - Hmmm...probably. Not sure. That's the most common example. What did you have in mind? –  PhD Jun 29 '12 at 6:27
See my answer regarding Bayesian rating systems below. Applied rating contexts typically have hundreds or thousands of objects being rated, and the aim is often to form the best estimate of the rating for the object given the available information. This is very different to a simple two group comparison as you might find say in an medical experiment with two groups. –  Jeromy Anglim Jun 29 '12 at 6:31

You can use a t-test to assess if there are differences in the means. The different sample sizes don't cause a problem for the t-test, and don't require the results to be interpreted with any extra care. Ultimately, you can even compare a single observation to an infinite population with a known distribution and mean and SD; for example someone with an IQ of 130 is smarter than 97.7% of people. One thing to note though, is that for a given $N$ (i.e., total sample size), power is maximized if the group $n$'s are equal; with highly unequal group sizes, you don't get as much additional resolution with each additional observation.

To clarify my point about power, here is a very simple simulation written for R:

set.seed(9)                            # this makes the simulation exactly reproducible

power5050 = vector(length=10000)       # these will store the p-values from each
power7525 = vector(length=10000)       # simulated test to keep track of how many
power9010 = vector(length=10000)       # are 'significant'

for(i in 1:10000){                     # I run the following procedure 10k times

n1a = rnorm(50, mean=0,  sd=1)       # I'm drawing 2 samples of size 50 from 2 normal
n2a = rnorm(50, mean=.5, sd=1)       # distributions w/ dif means, but = SDs

n1b = rnorm(75, mean=0,  sd=1)       # this version has group sizes of 75 & 25
n2b = rnorm(15, mean=.5, sd=1)

n1c = rnorm(90, mean=0,  sd=1)       # this one has 90 & 10
n2c = rnorm(10, mean=.5, sd=1)

power5050[i] = t.test(n1a, n2a, var.equal=T)$p.value # here t-tests are run & power7525[i] = t.test(n1b, n2b, var.equal=T)$p.value         # the p-values are stored
power9010[i] = t.test(n1c, n2c, var.equal=T)$p.value # for each version } sum(power5050<.05)/10000 # this code counts how many of the p-values for [1] 0.6888 # each of the versions are less than .05 & sum(power7525<.05)/10000 # divides the number by 10k to compute the % [1] 0.4219 # of times the results were 'significant'. That sum(power9010<.05)/10000 # gives an estimate of the power [1] 0.3206  Notice that in all cases$N=100$, but that in the first case$n_1=50$&$n_2=50$, in the second case$n_1=75$&$n_2=25$, and in the last case$n_1=90$and$n_2=10$. Note further that the standardized mean difference / data generating process was the same in all cases. However, whereas the test was 'significant' 69% of the time for the 50-50 sample, power was 42% with 75-25 and only 32% when the group sizes were 90-10. I think of this by analogy. If you want to know the area of a rectangle, and the perimeter is fixed, then the area will be maximized if the length and width are equal (i.e., if the rectangle is a square). On the other hand, as the length and width diverge (as the rectangle becomes elongated), the area shrinks. - power is maximized?? I'm not quite sure I understand. Could you please provide an example if possible? – PhD Jun 29 '12 at 6:33 The reason the t test can handle unequal sample sizes is that it takes account of the standard error of the estimates of the means for each group. That is the standard deviation of the group's distribution divided by the square root of the group's sample size. The goup with the much larger sample size will have the smaller standard error if the population standard deviations are bith equal or nearly so. – Michael Chernick Jun 29 '12 at 16:03 @gung - I'm not sure I really know which 'language' this simulation is written. I'm guessing 'R'? and I'm still trying to decipher it :) – PhD Jun 29 '12 at 18:53 The code is for R. I have commented it to make it easier to follow. You can just copy & paste it into R and run it yourself, if you have R; the set.seed() function will insure you get identical output. Let me know if it's still too difficult to follow. – gung Jun 29 '12 at 19:41 The analogy with area in this answer is not just suggestive of what's going on, it's exactly to the point. There's a very direct sense in which (given a fixed total sample size$N=n_1+n_2$- i.e. half the 'perimeter' of an$n_1\times n_2$rectangle), maximizing the product$n_1n_2\$ (the 'area') maximizes the precision of the estimate of the difference in means (and hence, power to identify it's not zero). It's algebraically trivial, so I won't labor the point further, but you couldn't have picked a more apt analogy. –  Glen_b Sep 18 at 1:17