I have an algorithm that for each sample $x_i$ returns an anomaly score $0<s_i<1$.
I use cross validation to set a threshold $th$ such that $x_i$ is anomalous if $s_i>th$.
During cross validation the threshold is set in order to have the desired False Positive Rate (FPR).
How many sample do I need to make sure that the threshold I have estimated will return exactly the desired FPR on new data?
For example if I have 10 samples in class 0 an 10 samples in class 1 I will not be able to estimate a FPR of $1\%$ as I would need at least 100 samples in class 0.
Is there a rule or a formula to know how many samples in class 0 do I need to reasonable estimate the threshold of a desired FPR?