63
$\begingroup$

Take the case of book ratings on a website. Book A is rated by 10,000 people with an average rating of 4.25 and the variance $\sigma = 0.5$. Similarly Book B is rated by 100 people and has a rating of 4.5 with $\sigma = 0.25$.

Now because of the large sample size of Book A the 'mean stabilized' to 4.25. Now for 100 people, it may be that if more people read Book B the mean rating may fall to 4 or 4.25.

  • how should one interpret the comparison of means from different samples and what are the best conclusions one can/should draw?

For example - can we really say Book B is better than Book A.

$\endgroup$
3
  • $\begingroup$ Are you specifically interested in the rating context? $\endgroup$ Jun 29, 2012 at 5:21
  • $\begingroup$ @JeromyAnglim - Hmmm...probably. Not sure. That's the most common example. What did you have in mind? $\endgroup$
    – PhD
    Jun 29, 2012 at 6:27
  • 2
    $\begingroup$ See my answer regarding Bayesian rating systems below. Applied rating contexts typically have hundreds or thousands of objects being rated, and the aim is often to form the best estimate of the rating for the object given the available information. This is very different to a simple two group comparison as you might find say in an medical experiment with two groups. $\endgroup$ Jun 29, 2012 at 6:31

2 Answers 2

74
$\begingroup$

You can use a t-test to assess if there are differences in the means. The different sample sizes don't cause a problem for the t-test, and don't require the results to be interpreted with any extra care. Ultimately, you can even compare a single observation to an infinite population with a known distribution and mean and SD; for example someone with an IQ of 130 is smarter than 97.7% of people. One thing to note though, is that for a given $N$ (i.e., total sample size), power is maximized if the group $n$'s are equal; with highly unequal group sizes, you don't get as much additional resolution with each additional observation.

To clarify my point about power, here is a very simple simulation written for R:

set.seed(9)                            # this makes the simulation exactly reproducible

power5050 = vector(length=10000)       # these will store the p-values from each 
power7525 = vector(length=10000)       # simulated test to keep track of how many 
power9010 = vector(length=10000)       # are 'significant'

for(i in 1:10000){                     # I run the following procedure 10k times

  n1a = rnorm(50, mean=0,  sd=1)       # I'm drawing 2 samples of size 50 from 2 normal
  n2a = rnorm(50, mean=.5, sd=1)       # distributions w/ dif means, but equal SDs

  n1b = rnorm(75, mean=0,  sd=1)       # this version has group sizes of 75 & 25
  n2b = rnorm(25, mean=.5, sd=1)

  n1c = rnorm(90, mean=0,  sd=1)       # this one has 90 & 10
  n2c = rnorm(10, mean=.5, sd=1)

  power5050[i] = t.test(n1a, n2a, var.equal=T)$p.value         # here t-tests are run &
  power7525[i] = t.test(n1b, n2b, var.equal=T)$p.value         # the p-values are stored
  power9010[i] = t.test(n1c, n2c, var.equal=T)$p.value         # for each version
}

mean(power5050<.05)                # this code counts how many of the p-values for
[1] 0.7019                         # each of the versions are less than .05 &
mean(power7525<.05)                # divides the number by 10k to compute the % 
[1] 0.5648                         # of times the results were 'significant'. That 
mean(power9010<.05)                # gives an estimate of the power
[1] 0.3261

Notice that in all cases $N=100$, but that in the first case $n_1=50$ & $n_2=50$, in the second case $n_1=75$ & $n_2=25$, and in the last case $n_1=90$ and $n_2=10$. Note further that the standardized mean difference / data generating process was the same in all cases. However, whereas the test was 'significant' 70% of the time for the 50-50 sample, power was 56% with 75-25 and only 33% when the group sizes were 90-10.

I think of this by analogy. If you want to know the area of a rectangle, and the perimeter is fixed, then the area will be maximized if the length and width are equal (i.e., if the rectangle is a square). On the other hand, as the length and width diverge (as the rectangle becomes elongated), the area shrinks.

$\endgroup$
4
  • $\begingroup$ power is maximized?? I'm not quite sure I understand. Could you please provide an example if possible? $\endgroup$
    – PhD
    Jun 29, 2012 at 6:33
  • 9
    $\begingroup$ The reason the t test can handle unequal sample sizes is that it takes account of the standard error of the estimates of the means for each group. That is the standard deviation of the group's distribution divided by the square root of the group's sample size. The goup with the much larger sample size will have the smaller standard error if the population standard deviations are bith equal or nearly so. $\endgroup$ Jun 29, 2012 at 16:03
  • 2
    $\begingroup$ The code is for R. I have commented it to make it easier to follow. You can just copy & paste it into R and run it yourself, if you have R; the set.seed() function will insure you get identical output. Let me know if it's still too difficult to follow. $\endgroup$ Jun 29, 2012 at 19:41
  • 12
    $\begingroup$ The analogy with area in this answer is not just suggestive of what's going on, it's exactly to the point. There's a very direct sense in which (given a fixed total sample size $N=n_1+n_2$ - i.e. half the 'perimeter' of an $n_1\times n_2$ rectangle), maximizing the product $n_1n_2$ (the 'area') maximizes the precision of the estimate of the difference in means (and hence, power to identify it's not zero). It's algebraically trivial, so I won't labor the point further, but you couldn't have picked a more apt analogy. $\endgroup$
    – Glen_b
    Sep 18, 2013 at 1:17
18
$\begingroup$

In addition to the answer mentioned by @gung referring you to the t-test, it sounds like you might be interested in Bayesian rating systems. Websites can use such systems to rank order items that vary in the number of votes received. Essentially, such systems work by assigning a rating that is a composite of the mean rating of all items plus the mean of the sample of ratings for the specific object. As the number of ratings increases, the weight assigned to the mean for the object increases and the weight assigned to mean rating of all items decreases. Perhaps check out bayesian averages.

Of course things can get a lot more complex as you deal with a wide range of issues such as voting fraud, changes over time, etc.

$\endgroup$
0

Not the answer you're looking for? Browse other questions tagged or ask your own question.