Determining sample size

If I have a population and want to draw samples from there, what are the steps to determine the number of samples to draw from the population? In other words, if I have population N and want to draw k samples from them so that the samples are representative of the entire population, how can I determine k?

I am having some insights from here but in the end of this document I find-

You can still use this formula if you don’t know your population standard deviation

which is my case (i.e., I do not know the population variance). How can I use that formula if I don't know the population standard deviation/ variance?

n.b. I posted this question to that forum but still no luck.

-
Rushdi, it then reads Although it’s unlikely that you know sigma when the population mean is not known, you may be able to determine sigma from ... a pilot test/simulation. Is that not clear? You can know sigma from 1) literature, 2) your pilot study, 3) special simulation processes. It is noteworthy that if you use some (previous study's) sample sigma it is better to pick the formula's Z from Student's rather than normal distribution. – ttnphns Jul 28 '12 at 5:37
@ttnphns, statistics is what I am learning for my research on natural language processing. So, please forgive me for being novice. Let me explain my situation: Two tools completely agree on categorizing 85k sentences (my entire population). Now, I want to take a sample and give that to two humans. This is to check whether the tools agreed genuinely or both of them together miscategorized the 85k sentences. Now, I need to know how many sentences should I pick as samples out of 85k sentences. – Rushdi Shams Jul 28 '12 at 6:02
Rushdi your problem is very different from what you ask in the question. I think you should post this your problem as a new question or completely rewritten current one. It will be a good question for this site. – ttnphns Jul 28 '12 at 6:08

3 Answers

There are two questions here and although I don't disagree with the comments I think the tone could be friendlier and a more helpful response could be given. (1) The first is a basic question from a beginner who wants to understand the puzzle of sample size determination. We should be honest and admit that it is a perplexing problem and the answer seems almost like circular reasoning.

(2) The second is a simple question with a simple answer.

Let me address (2) first since it is simple. In practice you never know the true variance. What you do is guess at it. This can be done by looking at how the answer changes as the population variance would vary over a range of plausible values. You might find the plausible values through literature on similar studies or through your ownpilot study or you can do a two stage adaptive design where the initial stage is used primarily to determine if you need additional data and if so how much. The first stage of an adaptive design is similar to a pilot study but the statistics behind it and the method of final determination of sample size is more complex.

Now about (1), determining sample size (or a formula for it) when comparing the difference between a hypothesized value of a parameter and the true value involves knowing something called the effect size. That is a normalized difference between the hypothesized value and the true value. But to know that requires knowing the true value. But that was your original problem! You want to find out or get a good estimate of the parameter. If you knew the true value you wouldn't have a problem and hence would not need to take a sample to estimate it. It is natural as a first reaction to see this circular argument that tells you that determining the sample size is impossible.

But that is not really so. What you really want to do is specify how big a difference between the hypothesized value and the true value (in terms of effect size) would be large enough for you to want to declare it significant. So you pretend that you have this effect size and see what the probability would be that you would reject the null hypothesis if you have this effect size. This probability will be a function of the sample size and the critical value for your test statistic. The critical value is determined by setting a significance level (aka type I error) for the test. This you can calculate. But it does depend on some unknowns that you have to guess at such as the population standard deviation that we spoke about in the answer to question (2).

This then describes sample size determination for hypothesis testing. a similar thing can be done for confidence intervals where effect size and type I error is replaced by a specified width and a confidence level respectively.

-

Some more thoughts to add to what Timo said. Usually you might follow some steps like

1. Decide how tight a precision you're trying to get. Say, you want to estimate the population mean to within 0.30.

2. Take a small pre-sample and estimate the standard deviation using

$s = \sqrt{ \frac{1}{n-1} \sum (x_i - \bar{x})^2 }$

3. Go use that formula you linked to, calculate n (and if n is small you might consider using a t table instead but since you're programming you probably have heaps of data).

4. Go and take that larger sample. Do what you wanted with it.

5. At this point, you might discover it wasn't enough data. What's the best course of action at this point really depends on what you're doing -- how important is it to get a precise result? How hard is getting more data? The statistical tools surrounding these questions are way more than I understand, and my advice to you would be to not worry too hard: if you feel like taking more data to get better precision, take more data, and if you don't, don't. A statistician would say it's technically incorrect to go back to step #2 here and just plug in your new, improved s, but... well, that's what I would do.

-
 thanks. Statistics is not what I do, so please forgive my naivety. =) if n is small you might consider using a t table instead : can you please explain why and how? – Rushdi Shams Jul 28 '12 at 6:24 Nice answer (+1) Indeed, you do not need to worry too much at that point. You need to get some data to know something about the variance, which in turn will tell you how much data you need. – gui11aume Jul 28 '12 at 14:29

I'm not an expert on the subject, but you can estimate the standard deviation either by calculating it from entire population (if it is feasible) or calculate it from a sample as large as possible. Size sqrt(population size) should be usually enough, but the larger, the better. Just use the following formula:

(taken from Wikipedia) to estimate unbiased standard deviation.

-
To calculate the entire population variance is not feasible for my case. And if I want to calculate the variance from a sample as large as possible- I need to know the sample size then which is again referring back to my question: how large the sample should be. The formula provided in the reference in my post says even if I don't know the population variance, the sample size can be calculated. I was baffled with this quote- how? Should I completely ignore the population variance? – Rushdi Shams Jul 28 '12 at 5:19
@Timo: That looks generally right except that the needed sample size shouldn't depend on the population size (many populations are infinite after all). For just a rough estimate of the variance, 30 or so should be good enough. Rushdi I've been reading a little on this topic recently and it actually gets pretty complicated; google "stopping condition" and see the kind of scary stuff you get... but if you just want to know about how much data to take, I'd say take a sample of 30, estimate the variance, collect more data, if it's not enough, repeat. – Owen Jul 28 '12 at 5:21
@Rushi and no, there's no way you could use that formula without some estimate of the variance to put into it; after all, n increases linearly with σ², so if σ² is off by a factor of 2, so is n. – Owen Jul 28 '12 at 5:24
You are determining sample size before you take a sample so what you do is what I said in my answer, you guess at, or use a pilot study to estimate it, or estimate it at the first stage of a two stage adaptive design. – Michael Chernick Jul 28 '12 at 19:42