# Number of samples required to estimate a desired False Positive Rate

I have an algorithm that for each sample $$x_i$$ returns an anomaly score $$0.

I use cross validation to set a threshold $$th$$ such that $$x_i$$ is anomalous if $$s_i>th$$.

During cross validation the threshold is set in order to have the desired False Positive Rate (FPR).

How many sample do I need to make sure that the threshold I have estimated will return exactly the desired FPR on new data?

For example if I have 10 samples in class 0 an 10 samples in class 1 I will not be able to estimate a FPR of $$1\%$$ as I would need at least 100 samples in class 0.

Is there a rule or a formula to know how many samples in class 0 do I need to reasonable estimate the threshold of a desired FPR?

• Honestly, it depends on the domain and context. For instance, the different sample size depends on the underlying statistical test between other conditions. Oct 8, 2019 at 15:04