2
$\begingroup$

I am doing a research study to find discussion about security defects in peer code review interactions. I have populated a database mined from code review repositories. Since, there are more than 200,000 threads in the database, I have created a list of keywords to query discussion threads that might be relevant to security discussion. Using the keywords, I filtered around 5000 threads. We manually inspect those threads and classify if any of those were relevant to security defect. We found that around 30% of those were indeed relevant to security defects. Now, I need to find out the effectiveness of our keyword set. For that, I plan to randomly select some threads, those do not contain any of the keywords. We will also manually inspect those to find how many of those have discussion about security defects. If very low (say less that 1%) of those are relevant to security discussion, we can conclude that our keywords set is effective. I need help in determining appropriate sample size (how many to inspect out of the 195,000 threads with no keywords) for judging the effectiveness of my keyword set. I have found that Cochran's formula can be used to determine population size for a survey. Can I apply that in this scenario?

$\endgroup$
2
$\begingroup$

If you mean Cochran's formula for sample size with proportions, yes you can use that. If you're expecting a probability of 0.01 and want precision ± half that then then you're going to need to look at about 1500.

But your guess on how precise you need to be is quite far off. If your keywords were very effective then you found all, or nearly all, of the discussions you want already. But you only flagged 0.75% of the total number of discussions, not even 1%. Therefore, if you found 1% of the remaining also were about security then that would not be a low number. It would instead show that there are more left than you found with the keywords. They're clearly effective to a degree (30% of your 5000), but they leave a substantial proportion behind.

I can't tell you what values to adjust toward but you're going to have to rethink what a low amount remaining is and come up with a good definition of "effective" before proceeding further. As a ballpark, in order to verify the effectiveness of the keywords I'm guessing your sample needs to be rather larger, in the 10,000 range.

$\endgroup$
  • $\begingroup$ I appreciate your help. I have so far randomly analyzed 200 threads and found that none of those had discussions about security. According to Yamane's simplified formula, if I take ±5% precision and 95% confidence interval, I need 400 samples. Do you think that if none of the 400 randomly selected threads were relevant to security will justify the effectiveness of my keywords? $\endgroup$ – amiangshu Aug 21 '13 at 21:13
  • $\begingroup$ See edits in the answer. 400 wasn't a big enough sample. $\endgroup$ – John Aug 26 '13 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.