I'm looking to do a third-party assessment of the false-positive rate of a video classification algorithm. Since I have a lot of video I'm trying to do a power analysis to figure out exactly how much video I need to look through so that it is representative of all the video data at a given confidence interval.
The algorithm flags video sequences that have at least one cat in it, and I'm looking to evaluate the frequency of false positives on a new unlabeled test set. So I have tagged all the video that my algorithm has identified a cat in and now want to sample the tagged video sections and look through them manually to validate my model since looking through all of it would take too long! Note, I'm not looking to refine the model at this point, just assess it.
My null hypothesis is that the FP rate of the sample of video I watch is equal to the FP rate of all the video.
I think I can use this formula to determine the number of video sequences to view:
Here is my question: am I thinking through this formulation correctly? Since my model has a CV false positive rate of ~0.96, I figure I can use that as a reference. Can I use that for the null hypothesis proportion, p0? Or will that be p, the true proportion?
I've been using this online calculator: http://powerandsamplesize.com/Calculators/Other/1-Sample-Binomial
I ask because when setting the parameters I have, I'm getting very small sample sizes, like less than 10 sequences to view. That can't be right.