# After training a binary classifier, are TPR & FPR independent of a test set?

Assume I trained a binary classifier $$f$$ and I was able to extract an optimal decision value $$t$$ such that the binary output $$\widehat{y}$$ of the classifier is: $$\widehat{y} = \begin{cases} 1, & f(x) \geq t \\ 0, & f(x) < t \end{cases}$$ Intrinsic to this classifier (at threshold $$t$$) are the True Positive Rate and False Positive Rate:

• $$TPR = \Pr(\widehat{y}=1|y=1)$$
• $$FPR = \Pr(\widehat{y}=1|y=0)$$

where $$y$$ is the true label of the sample.

Let's now assume that the classifier is deployed into a staging environment (call it $$S$$) for several weeks, where it will be presented with a fraction $$N = \Pr(y=0|S)$$ of negative and $$P = \Pr(y=1|S)$$ of positive examples.

My question is this: is it correct to expect the following fraction of positive classifications given the data in $$S$$? \begin{align} \Pr(\widehat{y}=1|S) &= \Pr(\widehat{y}=1|y=1,S) \cdot \Pr(y=1|S) \\ &+ \Pr(\widehat{y}=1|y=0,S) \cdot \Pr(y=0|S) \\ &= TPR \cdot P + FPR\cdot N \end{align}

That is, are $$TPR, FPR$$ independent of $$S$$?

• $$\Pr(\widehat{y}=1|y=1,S) = \Pr(\widehat{y}=1|y=1) = TPR$$
• $$\Pr(\widehat{y}=1|y=0,S) = \Pr(\widehat{y}=1|y=0) = FPR$$

My intuition is that this holds provided that the sample distribution in $$S$$ is similar to the hold-out test set used to compute $$TPR$$, but I need some help with formalizing this concept. Moreover, are there any other caveats?

• This is a correct application of the law of total probability to get the expected proportion of positive labels. But do you really know the proportion of positive & negative classes in S? And with the true proportion of positive classes known, you don't need to deploy the classifier to get the expectation, so what are you going to learn from the experiment? Commented Apr 18, 2022 at 21:20
• You're right, I overly simplified the problem. In my actual case, I only get to see $\Pr(\widehat{y}=1|S)$, and it is known that $N >> P$ (i.e. very few positive samples in $S$). Come to think of it, I guess my main concern is whether it's correct to use the $TPR$ computed on the test set as $\Pr(\widehat{y}=1|y=1,S)$, i.e. whether $TPR$ is really independent of $S$. My intuition is that $\Pr(\widehat{y}=1|y=1,S) = TPR$ provided that the sample distribution in $S$ is similar to the hold-out test set used to compute $TPR$. Commented Apr 18, 2022 at 21:39
• I think you are trying to claim that you will learn something from running the model in S but without doing any more labeling and just making assumptions. Say that the observed proportion of positive labels is much different than expected. You won't know what went wrong: data drift, model drift, bug in the pipeline.... Commented Apr 18, 2022 at 21:46
• What do you mean by 'expect'? Like the observed TPR and FPR are gonna be exactly the same for every $S$? Commented Apr 23, 2022 at 17:01
• Come to think of it, my main concern is the following: if I have TPR and FPR computed on a test set $S_1$, can I "reuse" these values to obtain an estimate of the total number of positive detections on a different test set $S_2$? i.e. does the following hold: $\Pr(\widehat{y}=1 | S_2) = TPR_{S_1} \cdot \Pr(y=1|S_2) + FPR_{S_1} \cdot \Pr(y=0|S_2)$? Commented Apr 26, 2022 at 7:45

Example Let's say you have a perfectly calibrated model (of the observations that are predicted to have 70% probability of being true, 70% of them will actually be true). You can have two datasets of N observations each that are both balanced (50% True). But, one dataset may be much more deterministic and all predictions are either 10% or 90%. The other may be more random and all predictions might be either 40% or 60%. Let's use a threshold of $$S=0.5$$. For the two datasets, the TPRs would be $$TPR_1 =\frac { \left(0*\frac{N}{2}*0.1\right) + \left(1*\frac{N}{2}*0.9\right) } {N/2}=0.9$$
$$TPR_2 =\frac { \left(0*\frac{N}{2}*0.4\right) + \left(1*\frac{N}{2}*0.6\right) } {N/2}=0.6$$
• So, to conclude, I guess that the TPR & FPR are not intrinsic to the classifier like I originally claimed -- they actually depend on the data the model is tested on. Furtermore, the probabilistic interpretation should always be conditioned on this data $\mathcal{D}$, e.g. $TPR({\mathcal{D}}) = \Pr(\widehat{y}=1|y=1,\mathcal{D})$. And, in general, $TPR({\mathcal{D}_1}) \neq TPR({\mathcal{D}_2})$ unless $\mathcal{D}_1$ and $\mathcal{D}_2$ are close. Is this correct? Commented Apr 19, 2022 at 21:36
• Disregarding calibration for a moment and just considering $y = 1$ when classifier output $> 0.5$, then the TPR will be the same for both datasets, i.e. it does not matter the actual value over or under the threshold, just the fact that you got a correct classification. Is something wrong with this argument? Commented Apr 20, 2022 at 7:35