How to determine appropriate sample size?

Question

I am doing a research study to find discussion about security defects in peer code review interactions. I have populated a database mined from code review repositories. Since, there are more than 200,000 threads in the database, I have created a list of keywords to query discussion threads that might be relevant to security discussion. Using the keywords, I filtered around 5000 threads. We manually inspect those threads and classify if any of those were relevant to security defect. We found that around 30% of those were indeed relevant to security defects. Now, I need to find out the effectiveness of our keyword set. For that, I plan to randomly select some threads, those do not contain any of the keywords. We will also manually inspect those to find how many of those have discussion about security defects. If very low (say less that 1%) of those are relevant to security discussion, we can conclude that our keywords set is effective. I need help in determining appropriate sample size (how many to inspect out of the 195,000 threads with no keywords) for judging the effectiveness of my keyword set. I have found that Cochran's formula can be used to determine population size for a survey. Can I apply that in this scenario?

Answer 1

If you mean Cochran's formula for sample size with proportions, yes you can use that. If you're expecting a probability of 0.01 and want precision ± half that then then you're going to need to look at about 1500.

But your guess on how precise you need to be is quite far off. If your keywords were very effective then you found all, or nearly all, of the discussions you want already. But you only flagged 0.75% of the total number of discussions, not even 1%. Therefore, if you found 1% of the remaining also were about security then that would not be a low number. It would instead show that there are more left than you found with the keywords. They're clearly effective to a degree (30% of your 5000), but they leave a substantial proportion behind.

I can't tell you what values to adjust toward but you're going to have to rethink what a low amount remaining is and come up with a good definition of "effective" before proceeding further. As a ballpark, in order to verify the effectiveness of the keywords I'm guessing your sample needs to be rather larger, in the 10,000 range.