3
$\begingroup$

I have an algorithm that for each sample $x_i$ returns an anomaly score $0<s_i<1$.

I use cross validation to set a threshold $th$ such that $x_i$ is anomalous if $s_i>th$.

During cross validation the threshold is set in order to have the desired False Positive Rate (FPR).

How many sample do I need to make sure that the threshold I have estimated will return exactly the desired FPR on new data?

For example if I have 10 samples in class 0 an 10 samples in class 1 I will not be able to estimate a FPR of $1\%$ as I would need at least 100 samples in class 0.

Is there a rule or a formula to know how many samples in class 0 do I need to reasonable estimate the threshold of a desired FPR?

Improve this question
$\endgroup$
1
  • $\begingroup$ Honestly, it depends on the domain and context. For instance, the different sample size depends on the underlying statistical test between other conditions. $\endgroup$
    – Francisco Maria Calisto
    Oct 8, 2019 at 15:04

2 Answers 2

Reset to default
1
$\begingroup$

Is this question simply a re-framing of a power analysis question? Seems like you would answer this by calculating a sample size for a desired alpha, which in itself is an a priori "threshold" acceptable to a given researcher (typically 0.05 in social science).

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ sorry I do not understand. Can you provide more details? $\endgroup$
    – Donbeo
    Jul 11, 2018 at 15:49
  • 2
    $\begingroup$ Power analysis allows “to determine the sample size required to detect an effect of a given size with a given degree of confidence” (Clay Ford). If you use R, package pwr may be of help. More information on what power analysis is, how can it help you, and how to perform it in R is here: cran.r-project.org/web/packages/pwr/vignettes/pwr-vignette.html $\endgroup$
    – Marco Plebani
    Jul 11, 2018 at 16:20
0
$\begingroup$

In theory [1], the number you need to approach the level of significance for you problem. Nevertheless, consider to include both significance and power levels. Also, consider the potential impact of your FPR results as it may directly change your appropriate level of significance.

The level of significance (referred to as α) defines the strength of identifying an effect when no effect exists, in other words having a false-positive result. A type I error (false-positive) occurs when we wrongly conclude there is a difference, i.e. with an α of 0.05 there is a 5% risk of a false-positive result. The lower the level of α the less likely it is that a type I error will occur. When determining the appropriate level of significance it will be necessary to consider the potential impact of a false-positive result; if the potential impact is serious then a lower level of significance, for example, α = 0.01 (1% risk), might be selected.

[1] Burmeister, E. and Aitken, L.M., 2012. Sample size: How many is enough?. Australian Critical Care, 25(4), pp.271-274.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.