# Numbers 1-100, calculating how many I must pick to get x% of reliability

I would like to know the formula or method of calculating this.

For example, I want to know how many numbers I must pick in order to have 95% confidence that I will be within 2% of the average.

I'm also interested in doing a similiar analysis for the median as well.

Update: Numbers are not put back in the pool. I think this has to do with combinations rather than trials? For example, if I pick 1 number or 99 numbers there are 100 combinations, if I pick 100 numbers there is 1 combination etc. I think you could figure out all of the combinations when you pick 1 ball, 2 balls, 3 balls etc and see how many are within 2% of the median.

Im interested in seeing just how reliable the average price statistic is for the housing market. For example, in a single month 5% of the houses may sell, or maybe 20% will. Housing prices are all over the map but I wanted to do something simplistic like this.

And now I have thought of something else...let's say I have 1-500 instead of 1-100. If I pick from 1-100 and pick 10 balls(10% of them), Im guessing that the answer would differ from taking 50 balls out of 1-500(10% of them as well)?

-
I'm not sure what this means; do you have the numbers 1-100 in a bag, and are going to pick them out one at a time without replacement? –  shabbychef Sep 27 '10 at 18:05
@shabbychef: The formula in that case is n = 0, because the average and median are both 50.5! :-) –  whuber Sep 27 '10 at 18:13
@whuber: yes, this is why I'm not sure what is being asked. although the question could be: how many numbers must I pick until I pick one of 49,50,51 or 52, with 95% probability... –  shabbychef Sep 27 '10 at 18:32
@shabbychef: I completely agree with you; my comment was just an amusing way to emphasize the vagueness. The answers are obvious--sequential methods or standard experimental design methods, with a reference to the literature on the relative power of robust methods to handle the case of the median--but exactly which answer is appropriate (or best) will depend on how the numbers are being obtained and assumptions about their possible distribution. E.g., if the numbers are in the range $[1,100]$ and that's all you know, I can guarantee 2% relative error with 95% confidence in 9228 draws. –  whuber Sep 27 '10 at 20:23
@shabbychef: Yep, the ambiguity is certainly there. The followup about the median could make for some difficult homework, though: you have to make a lot of assumptions and the analysis is not found in elementary courses. (Under some distributional assumptions the median is a more efficient estimator of central tendency than the mean, but most of the time in practice it's less efficient.) –  whuber Sep 27 '10 at 21:47

I did a little simulation in R, which resulted in 98. If you go to 5% of the mean then you're down to 93 and if you go to 10% then it's 76. Similar results are obtained for the median.

The code I used:

numbers <- 1:100
numTrials <- 1000

means <- matrix(nrow = numTrials, ncol = length(numbers))

for (n in 1:length(numbers))
{
for (t in 1:numTrials)
{
vals <- sample(numbers, n)
means[t, n] <- mean(vals) # or median if you want the median
}
}

realAvg <- mean(numbers)
uplim <- realAvg + 0.05*realAvg
downlim <- realAvg - 0.05*realAvg

res <- unlist(apply(means, 2, function(x){
length(which(x>downlim & x<uplim))/length(x)}))
plot(1:length(numbers), res*100, t="l", xlab="Numbers picked",
ylab="% trials with mean within limits")
abline(h=95, col="red")

print(which(res>=0.95))


PS: anyone would like to explain the "sawtooth" look of the graph?

-
You assumed a uniform distribution, so why do a simulation? An exact answer is straightforward to obtain. (Indeed, my facetious comment still applies: by assuming a uniform, you know a priori that the mean is 50.5, so you don't need to pick any numbers at all!) But if all you know is that the numbers are in the range [1,100], you have to plan for the worst case, which will require up to 9000+ draws (when half the values are 1 and half are 100). First we need to hear from the OP concerning what assumptions are valid to make here. –  whuber Sep 27 '10 at 20:30
@whuber, I assumed the numbers where NOT put back in the "pool" (that is why with 100 draws you have 100% of the trials giving the good result). Also, the simulation works for ANY set of 100 numbers, I just used 1:100 because that's what the question asked (well, I guess at least), but it can be any set of 100 numbers :) –  nico Sep 27 '10 at 20:49
Anyway, you can easily modify the script to allow for resampling the same number multiple times (add replace=TRUE in the sample call). The result turns out to be around 5000 in that case –  nico Sep 27 '10 at 21:06
@nico: Now you're getting to the heart of the matter: the trick is to find a method that works for the full range of possible pool contents consistent with one's assumptions. All a simulation can do is indicate what can be achieved by sampling when one particular assumption is true. Your modification already reveals what's at stake: the difference between a sample of 98 and one of 5000 is enormous. This suggests looking a Bayesian and/or sequential sampling methods instead of designing a fixed sample size based on a frequentist assumption. –  whuber Sep 27 '10 at 22:09
@whuber: OK, I understand what you're saying now. Actually, to clarify on my previous comment: the code works for any 100 number set, but it does not give the same result. If we only use numbers between 90 and 100, for instance, 2 or 3 draws are enough to estimate the mean. Still, an interesting problem –  nico Sep 28 '10 at 7:03

I guess that you are looking for experimental design for estimating the mean. I am not aware of something similar for the median though.

-
The asymptotic distribution of the median is Normal with variance inversely proportional to the square of the pdf at the median. Thus, experimental designs for median estimation have to make some specific assumptions about the nature of the distribution near its middle. (See mathworld.wolfram.com/StatisticalMedian.html for example.) Usually people study a range of parametric alternatives to compare the efficiency of the median to that of the mean; relative efficiencies translate via the usual square-root law into relative sample sizes. –  whuber Sep 28 '10 at 15:35
numbers <- 1:500
numTrials <- 1000

means <- matrix(nrow = numTrials, ncol = length(numbers))

for (n in 1:length(numbers))
{
for (t in 1:numTrials)
{
vals <- sample(numbers, n)
means[t, n] <- median(vals)
}
}

realAvg <- median(numbers)
uplim <- realAvg + 0.05*realAvg
downlim <- realAvg - 0.05*realAvg

res <- unlist(apply(means, 2, function(x){
length(which(x>downlim & x<uplim))/length(x)}))
plot(1:length(numbers), res*500, t="l", xlab="Numbers picked",
ylab="% trials with mean within limits")
abline(h=95, col="red")

print(which(res>=0.95))

-

A couple of points first:
1) The answer will be very large, so we are safe to assume the Central Limit Theorem is in operation, even if the distribution is all 1s and 100s (i.e. Bernoulli).
2) We can use normal deviates (z-scores) rather than t-tests because the required N will be very large - this is a simplification.
3. If we are sampling without replacement the problem statement seems to have only one interpretation - the numbers 1,...,100 - a uniform discrete distribution. whuber and nico have already provided answers, so I will deal with sampling with replacement and an undefined probability mass function (PMF).

Approach 1:
Assume a discrete uniform distribution and sampling with replacement. The mean is 50.5, the SD is sqrt((n^2 - 1)/12) ~ 29

We want a maximum error of 2% so delta = abs(est_mean - true_mean)/true_mean = 1.01

Assume a normal distribution, the standard error of the mean is: \begin{align} SEM &= SD/sqrt(n)\\ sqrt(n) &= SD/SEM\\ n &= (SD/SEM)^2\\ n &= (SD/(delta/1.96))^2\\ n &= (29/(1.01/1.96))^2\\ n &\approx 3167\\ \end{align}

Approach 2:
Assume the worst case. I'm pretty sure this is a 50:50 split in the pmf between 1 and 100. If so our mean is 50.5 and our SD is sqrt(49.5^2)=49.5.

\begin{align} SEM &= SD/sqrt(n)\\ sqrt(n) &= SD/SEM\\ n &= (SD/SEM)^2\\ n &= (SD/(delta/1.96))^2\\ n &= (49.5/(1.01/1.96))^2\\ n &\approx 9227\\ \end{align}

I'll let someone else do the sample sizes for medians.

-
The answer of 25000, which is obviously ridiculous no matter what the true state of affairs may be, indicates that a power calculation is not appropriate here. The answer of 9228 I obtained guarantees that no matter what the distribution may be (provided only that its values all lie between 1 and 100), 95% of all simple random samples (without replacement) with n=9228 will lie within 2% of the mean. Power is not relevant! –  whuber Sep 28 '10 at 15:31
Conceded whuber. I'll modify the answer to reflect this, although the question appears to have changed in the interim. BTW it's "with replacement". –  Thylacoleo Sep 29 '10 at 2:31
Thanks about the BTW. I mis-typed and intended to say with replacement. –  whuber Sep 29 '10 at 14:33