I am doing a research study to find discussion about security defects in peer code review interactions. I have populated a database mined from code review repositories. Since, there are more than 200,000 threads in the database, I have created a list of keywords to query discussion threads that might be relevant security discussion. Using the keywords, I filtered around 5000 threads. We manually inspect those threads and classify if any of those were relevant to security defect. We found that around 30% of those were indeed relevant to security defects. Now, I need to find out the effectiveness of our keyword set. For that, I plan to randomly select some threads, those do not contain any of the keywords. We will also manually inspect those to find how many of those have discussion about security defects. If very low (say less that 1%) of those are relevant to security discussion, we can conclude that our keywords set is effective. I need help in determining appropriate sample size (how many to inspect out of the 190,000 threads with no keywords) for judging the effectiveness of my keyword set. I have found that Cochran's formula can be used to determine population size for a survey. Can I apply that in this scenerio?

I am doing a research study to find discussion about security defects in peer code review interactions. I have populated a database mined from code review repositories. Since, there are more than 200,000 threads in the database, I have created a list of keywords to query discussion threads that might be relevant to security discussion. Using the keywords, I filtered around 5000 threads. We manually inspect those threads and classify if any of those were relevant to security defect. We found that around 30% of those were indeed relevant to security defects. Now, I need to find out the effectiveness of our keyword set. For that, I plan to randomly select some threads, those do not contain any of the keywords. We will also manually inspect those to find how many of those have discussion about security defects. If very low (say less that 1%) of those are relevant to security discussion, we can conclude that our keywords set is effective. I need help in determining appropriate sample size (how many to inspect out of the 195,000 threads with no keywords) for judging the effectiveness of my keyword set. I have found that Cochran's formula can be used to determine population size for a survey. Can I apply that in this scenario?