# Determining sample size for ML algorithm validation

I'm looking to do a third-party assessment of the false-positive rate of a video classification algorithm. Since I have a lot of video I'm trying to do a power analysis to figure out exactly how much video I need to look through so that it is representative of all the video data at a given confidence interval.

The algorithm flags video sequences that have at least one cat in it, and I'm looking to evaluate the frequency of false positives on a new unlabeled test set. So I have tagged all the video that my algorithm has identified a cat in and now want to sample the tagged video sections and look through them manually to validate my model since looking through all of it would take too long! Note, I'm not looking to refine the model at this point, just assess it.

My null hypothesis is that the FP rate of the sample of video I watch is equal to the FP rate of all the video.

I think I can use this formula to determine the number of video sequences to view:

Here is my question: am I thinking through this formulation correctly? Since my model has a CV false positive rate of ~0.96, I figure I can use that as a reference. Can I use that for the null hypothesis proportion, p0? Or will that be p, the true proportion?

I've been using this online calculator: http://powerandsamplesize.com/Calculators/Other/1-Sample-Binomial

I ask because when setting the parameters I have, I'm getting very small sample sizes, like less than 10 sequences to view. That can't be right.

Sample size calculation is a statistical consideration to determine the precision and power of a particular analysis where data comprise a random sample.

Validation (of an algorithm) involves testing performance under a variety of non-random scenarios, and describing any deficiencies or updating the algorithm as needed. The number of scenarios is determined by the scope of the algorithm.

If you randomly sample a bunch of videos and want to run your algorithm to estimate the proportion of cats in each video, that is a statistical analysis. The proportion $$p$$ is a useless quantity, because the videos comprise a convenience sample and you lack a gold standard. The test is even more useless because you don't actually have a hypothesis.

A gold standard means a viewer carefully watches the video and declares whether or not it has a cat in it.

If you want to describe the interrater agreement, use a test of Cohen's Kappa. This will give a powerful test for how often the algorithm agrees with the viewer, independent of the actual baseline frequency of cats.

• here the null hypothesis is: the rate of video sequences containing cats in the population is equal to the rate of video sequences containing cats in the sample. The "gold standard" that you speak of is exactly what I'm trying to do here, I want to carefully watch a sample of the videos (hopefully not all of them) and extrapolate my results at a 95% type I error to the population. Any ideas? – Zafar Jun 5 at 19:00
• @Zafar that is calibration only. Calibration alone does not a good classifier make. – AdamO Jun 5 at 19:15
• I definitely appreciate the expert advice here. For the example I posted, which is a simplification of my actual problem, let's take the classifier as final. I'm simply trying to assess the false positive rate using new video data. Think of it as a third-part verification of the FP rate. Is the method that I laid out above misguided? – Zafar Jun 5 at 20:20
• My goal is to be able to say, "looking at a sample of the video, the FP rate is x given a confidence/precision of y". – Zafar Jun 6 at 2:34