49
$\begingroup$

Take the case of book ratings on a website. Book A is rated by 10,000 people with an average rating of 4.25 and the variance $\sigma = 0.5$. Similarly Book B is rated by 100 people and has a rating of 4.5 with $\sigma = 0.25$.

Now because of the large sample size of Book A the 'mean stabilized' to 4.25. Now for 100 people, it may be that if more people read Book B the mean rating may fall to 4 or 4.25.

  • how should one interpret the comparison of means from different samples and what are the best conclusions one can/should draw?

For example - can we really say Book B is better than Book A.

$\endgroup$
  • $\begingroup$ Are you specifically interested in the rating context? $\endgroup$ – Jeromy Anglim Jun 29 '12 at 5:21
  • $\begingroup$ @JeromyAnglim - Hmmm...probably. Not sure. That's the most common example. What did you have in mind? $\endgroup$ – PhD Jun 29 '12 at 6:27
  • 2
    $\begingroup$ See my answer regarding Bayesian rating systems below. Applied rating contexts typically have hundreds or thousands of objects being rated, and the aim is often to form the best estimate of the rating for the object given the available information. This is very different to a simple two group comparison as you might find say in an medical experiment with two groups. $\endgroup$ – Jeromy Anglim Jun 29 '12 at 6:31
57
$\begingroup$

You can use a t-test to assess if there are differences in the means. The different sample sizes don't cause a problem for the t-test, and don't require the results to be interpreted with any extra care. Ultimately, you can even compare a single observation to an infinite population with a known distribution and mean and SD; for example someone with an IQ of 130 is smarter than 97.7% of people. One thing to note though, is that for a given $N$ (i.e., total sample size), power is maximized if the group $n$'s are equal; with highly unequal group sizes, you don't get as much additional resolution with each additional observation.

To clarify my point about power, here is a very simple simulation written for R:

set.seed(9)                            # this makes the simulation exactly reproducible

power5050 = vector(length=10000)       # these will store the p-values from each 
power7525 = vector(length=10000)       # simulated test to keep track of how many 
power9010 = vector(length=10000)       # are 'significant'

for(i in 1:10000){                     # I run the following procedure 10k times

  n1a = rnorm(50, mean=0,  sd=1)       # I'm drawing 2 samples of size 50 from 2 normal
  n2a = rnorm(50, mean=.5, sd=1)       # distributions w/ dif means, but equal SDs

  n1b = rnorm(75, mean=0,  sd=1)       # this version has group sizes of 75 & 25
  n2b = rnorm(25, mean=.5, sd=1)

  n1c = rnorm(90, mean=0,  sd=1)       # this one has 90 & 10
  n2c = rnorm(10, mean=.5, sd=1)

  power5050[i] = t.test(n1a, n2a, var.equal=T)$p.value         # here t-tests are run &
  power7525[i] = t.test(n1b, n2b, var.equal=T)$p.value         # the p-values are stored
  power9010[i] = t.test(n1c, n2c, var.equal=T)$p.value         # for each version
}

mean(power5050<.05)                # this code counts how many of the p-values for
[1] 0.7019                         # each of the versions are less than .05 &
mean(power7525<.05)                # divides the number by 10k to compute the % 
[1] 0.5648                         # of times the results were 'significant'. That 
mean(power9010<.05)                # gives an estimate of the power
[1] 0.3261

Notice that in all cases $N=100$, but that in the first case $n_1=50$ & $n_2=50$, in the second case $n_1=75$ & $n_2=25$, and in the last case $n_1=90$ and $n_2=10$. Note further that the standardized mean difference / data generating process was the same in all cases. However, whereas the test was 'significant' 70% of the time for the 50-50 sample, power was 56% with 75-25 and only 33% when the group sizes were 90-10.

I think of this by analogy. If you want to know the area of a rectangle, and the perimeter is fixed, then the area will be maximized if the length and width are equal (i.e., if the rectangle is a square). On the other hand, as the length and width diverge (as the rectangle becomes elongated), the area shrinks.

$\endgroup$
  • $\begingroup$ power is maximized?? I'm not quite sure I understand. Could you please provide an example if possible? $\endgroup$ – PhD Jun 29 '12 at 6:33
  • 5
    $\begingroup$ The reason the t test can handle unequal sample sizes is that it takes account of the standard error of the estimates of the means for each group. That is the standard deviation of the group's distribution divided by the square root of the group's sample size. The goup with the much larger sample size will have the smaller standard error if the population standard deviations are bith equal or nearly so. $\endgroup$ – Michael Chernick Jun 29 '12 at 16:03
  • $\begingroup$ @gung - I'm not sure I really know which 'language' this simulation is written. I'm guessing 'R'? and I'm still trying to decipher it :) $\endgroup$ – PhD Jun 29 '12 at 18:53
  • 2
    $\begingroup$ The code is for R. I have commented it to make it easier to follow. You can just copy & paste it into R and run it yourself, if you have R; the set.seed() function will insure you get identical output. Let me know if it's still too difficult to follow. $\endgroup$ – gung - Reinstate Monica Jun 29 '12 at 19:41
  • 8
    $\begingroup$ The analogy with area in this answer is not just suggestive of what's going on, it's exactly to the point. There's a very direct sense in which (given a fixed total sample size $N=n_1+n_2$ - i.e. half the 'perimeter' of an $n_1\times n_2$ rectangle), maximizing the product $n_1n_2$ (the 'area') maximizes the precision of the estimate of the difference in means (and hence, power to identify it's not zero). It's algebraically trivial, so I won't labor the point further, but you couldn't have picked a more apt analogy. $\endgroup$ – Glen_b Sep 18 '13 at 1:17
10
$\begingroup$

In addition to the answer mentioned by @gung referring you to the t-test, it sounds like you might be interested in Bayesian rating systems (e.g., here's a discussion). Websites can use such systems to rank order items that vary in the number of votes received. Essentially, such systems work by assigning a rating that is a composite of the mean rating of all items plus the mean of the sample of ratings for the specific object. As the number of ratings increases, the weight assigned to the mean for the object increases and the weight assigned to mean rating of all items decreases. Perhaps check out bayesian averages.

Of course things can get a lot more complex as you deal with a wide range of issues such as voting fraud, changes over time, etc.

$\endgroup$
  • $\begingroup$ Sweet. Never heard of it. I'll definitely look into it. Maybe that's what I'm after, after all :) $\endgroup$ – PhD Jun 29 '12 at 6:35

protected by kjetil b halvorsen Nov 12 '17 at 17:17

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.