Number of samples required to estimate a desired False Positive Rate I have an algorithm that for each sample $x_i$ returns an anomaly score $0<s_i<1$.  
I use cross validation to set a threshold $th$ such that $x_i$ is anomalous if $s_i>th$. 
During cross validation the threshold is set in order to have the desired False Positive Rate (FPR).   
How many sample do I need to make sure that the threshold I have estimated will return exactly the desired FPR on new data?
For example if I have 10 samples in class 0 an 10 samples in class 1 I will not be able to estimate a FPR of $1\%$  as I would need at least 100 samples in class 0. 
Is there a rule or a formula to know how many samples in class 0 do I need to reasonable estimate the threshold of a desired FPR? 
 A: Is this question simply a re-framing of a power analysis question? Seems like you would answer this by calculating a sample size for a desired alpha, which in itself is an a priori "threshold" acceptable to a given researcher (typically 0.05 in social science).
A: In theory [1], the number you need to approach the level of significance for you problem. Nevertheless, consider to include both significance and power levels. Also, consider the potential impact of your FPR results as it may directly change your appropriate level of significance.

The level of significance (referred to as α) defines the strength of identifying an effect when no effect exists, in other words having a false-positive result. A type I error (false-positive) occurs when we wrongly conclude there is a difference, i.e. with an α of 0.05 there is a 5% risk of a false-positive result. The lower the level of α the less likely it is that a type I error will occur. When determining the appropriate level of significance it will be necessary to consider the potential impact of a false-positive result; if the potential impact is serious then a lower level of significance, for example, α = 0.01 (1% risk), might be selected.

[1] Burmeister, E. and Aitken, L.M., 2012. Sample size: How many is enough?. Australian Critical Care, 25(4), pp.271-274.
