4
$\begingroup$

I have the following statistical problem:

I have two boxes, each has exactly 10000 numbers in it. Each number is either 0, 1, or 3. What I need to know is if the average amount in one box is bigger or smaller than the average amount in the other box.

Unfortunately, picking numbers is very costly for me. So what I would like to know is, how and how many numbers do I have to pick so that I can say with e.g. 90% probability that one box's average is below or above the other ones'?

EDIT:

I can iteratively randomly pick one number of a chosen box, and after each pick I can recalculate the probability. So I do not want to calculate how many I have to pick beforhand, but I would like to update a probability after each pick.

For example, when I randomly pick 10 numbers out of box 1 and each time I get a "3", and from the other box I pick 10 numbers and each one is "1", chances should be high that box 1's mean is higher than that of box 2. So I think it should be possible to somehow calculate if it is useful to continue picking or say with high probability that further picking will not change the final outcome.

This is no homework, but should be part of an optimization software I am writing. I want to quickly evaluate if one solution is better than another one.

Improve this question
$\endgroup$
10
  • 2
    $\begingroup$ To give you some intuition, consider two scenarios. (1) Both boxes have 9999 1's; one has one 0 and another has one 3. (2) One box contains only 0's and the other contains only 3's. In scenario (1), you will have to pick all 10,000 numbers from each box before you can identify the one with the higher average; in (2), most routine tests will fail but the answer will be obvious after a half dozen draws from each box. Given such a range of possibilities, can you say anything more about what might be in your two boxes? $\endgroup$
    – whuber ♦
    Aug 10, 2011 at 19:34
  • 2
    $\begingroup$ as @whuber points out, you need to know something about the posited 'effect size'. Here that would be defined as $\Delta =(\mu_1 - \mu_2) / \sigma$, where $\mu_i$ is the mean value in box $i$, and $\sigma$ is the pooled standard deviation. Given this, the sample size should be about $24 / \Delta^2$, with half of these drawn from each box. This should give you a 2-sided 2-sample t-test with 0.05 type I rate and 90 percent power assuming the effect size. $\endgroup$
    – shabbychef
    Aug 10, 2011 at 20:18
  • $\begingroup$ Is this a homework question? $\endgroup$
    – Iterator
    Aug 11, 2011 at 1:16
  • 1
    $\begingroup$ @whuber This question is straight out of intro hypothesis testing. Is there a reason to address it as insoluble? $\endgroup$
    – Iterator
    Aug 11, 2011 at 1:22
  • $\begingroup$ @Iterator: This may be a question straight out of stochastic process, namely, <a href="en.wikipedia.org/wiki/Two-armed_bandit">two-arm bandit problems</a> (and that's what the OP would want to look up -- this may be at least a partial answer). In my graduate level stochastic processes class, we did not cover it. $\endgroup$
    – StasK
    Aug 11, 2011 at 6:20

1 Answer 1

Reset to default
3
$\begingroup$

Your initial question was a simple problem of comparing means via one batch sample. Your revised question is about sequential analysis, which is even more judicious about sample size. I wish more people thought about sequential analysis, as it's much more cost-effective when samples are expensive.

In deference to the "teach a man to fish" school of thought, I'll recommend looking into a book on this topic. Wald's 1973 book is very nice, though I've never seen a Dover book list for so much on Amazon.

Improve this answer
$\endgroup$
3
  • $\begingroup$ thanks a lot, I have never heard about "sequential analysis" :) $\endgroup$
    – martinus
    Aug 11, 2011 at 12:58
  • $\begingroup$ Good luck. A book like Wald's is good for considering different scenarios and hypotheses; I'm sure some of the newer books are very good, too. It's also a good idea to simulate your sampling and testing plan, just to be sure - debugging it live when samples are expensive is a bad idea. $\endgroup$
    – Iterator
    Aug 11, 2011 at 13:31
  • $\begingroup$ +1 That's exactly the right approach. But given the straightforward nature of the problem, couldn't you at least sketch the nature of the solution, if not give the solution itself? $\endgroup$
    – whuber ♦
    Aug 11, 2011 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.