5
$\begingroup$

Assume I trained a binary classifier $f$ and I was able to extract an optimal decision value $t$ such that the binary output $\widehat{y}$ of the classifier is: $$ \widehat{y} = \begin{cases} 1, & f(x) \geq t \\ 0, & f(x) < t \end{cases} $$ Intrinsic to this classifier (at threshold $t$) are the True Positive Rate and False Positive Rate:

  • $TPR = \Pr(\widehat{y}=1|y=1)$
  • $FPR = \Pr(\widehat{y}=1|y=0)$

where $y$ is the true label of the sample.

Let's now assume that the classifier is deployed into a staging environment (call it $S$) for several weeks, where it will be presented with a fraction $N = \Pr(y=0|S)$ of negative and $P = \Pr(y=1|S)$ of positive examples.

My question is this: is it correct to expect the following fraction of positive classifications given the data in $S$? $$ \begin{align} \Pr(\widehat{y}=1|S) &= \Pr(\widehat{y}=1|y=1,S) \cdot \Pr(y=1|S) \\ &+ \Pr(\widehat{y}=1|y=0,S) \cdot \Pr(y=0|S) \\ &= TPR \cdot P + FPR\cdot N \end{align} $$

That is, are $TPR, FPR$ independent of $S$?

  • $\Pr(\widehat{y}=1|y=1,S) = \Pr(\widehat{y}=1|y=1) = TPR$
  • $\Pr(\widehat{y}=1|y=0,S) = \Pr(\widehat{y}=1|y=0) = FPR$

My intuition is that this holds provided that the sample distribution in $S$ is similar to the hold-out test set used to compute $TPR$, but I need some help with formalizing this concept. Moreover, are there any other caveats?

Improve this question
$\endgroup$
5
  • $\begingroup$ This is a correct application of the law of total probability to get the expected proportion of positive labels. But do you really know the proportion of positive & negative classes in S? And with the true proportion of positive classes known, you don't need to deploy the classifier to get the expectation, so what are you going to learn from the experiment? $\endgroup$
    – dipetkov
    Commented Apr 18, 2022 at 21:20
  • $\begingroup$ You're right, I overly simplified the problem. In my actual case, I only get to see $\Pr(\widehat{y}=1|S)$, and it is known that $N >> P$ (i.e. very few positive samples in $S$). Come to think of it, I guess my main concern is whether it's correct to use the $TPR$ computed on the test set as $\Pr(\widehat{y}=1|y=1,S)$, i.e. whether $TPR$ is really independent of $S$. My intuition is that $\Pr(\widehat{y}=1|y=1,S) = TPR$ provided that the sample distribution in $S$ is similar to the hold-out test set used to compute $TPR$. $\endgroup$
    – Alexandru Dinu
    Commented Apr 18, 2022 at 21:39
  • 2
    $\begingroup$ I think you are trying to claim that you will learn something from running the model in S but without doing any more labeling and just making assumptions. Say that the observed proportion of positive labels is much different than expected. You won't know what went wrong: data drift, model drift, bug in the pipeline.... $\endgroup$
    – dipetkov
    Commented Apr 18, 2022 at 21:46
  • $\begingroup$ What do you mean by 'expect'? Like the observed TPR and FPR are gonna be exactly the same for every $S$? $\endgroup$
    – Sextus Empiricus
    Commented Apr 23, 2022 at 17:01
  • $\begingroup$ Come to think of it, my main concern is the following: if I have TPR and FPR computed on a test set $S_1$, can I "reuse" these values to obtain an estimate of the total number of positive detections on a different test set $S_2$? i.e. does the following hold: $\Pr(\widehat{y}=1 | S_2) = TPR_{S_1} \cdot \Pr(y=1|S_2) + FPR_{S_1} \cdot \Pr(y=0|S_2)$? $\endgroup$
    – Alexandru Dinu
    Commented Apr 26, 2022 at 7:45

1 Answer 1

Reset to default
1
$\begingroup$

Nope. Not really sure you can conclude anything useful here. If you think about your model as assigning probabilities of observations being true then I think the issue is more clear.

Example Let's say you have a perfectly calibrated model (of the observations that are predicted to have 70% probability of being true, 70% of them will actually be true). You can have two datasets of N observations each that are both balanced (50% True). But, one dataset may be much more deterministic and all predictions are either 10% or 90%. The other may be more random and all predictions might be either 40% or 60%. Let's use a threshold of $S=0.5$. For the two datasets, the TPRs would be $$ TPR_1 =\frac { \left(0*\frac{N}{2}*0.1\right) + \left(1*\frac{N}{2}*0.9\right) } {N/2}=0.9$$

$$ TPR_2 =\frac { \left(0*\frac{N}{2}*0.4\right) + \left(1*\frac{N}{2}*0.6\right) } {N/2}=0.6$$

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for the insight. When you say one dataset may be much more deterministic does this mean that the examples are easier to classify (i.e. there's less uncertainty from the classifier)? $\endgroup$
    – Alexandru Dinu
    Commented Apr 19, 2022 at 21:27
  • $\begingroup$ So, to conclude, I guess that the TPR & FPR are not intrinsic to the classifier like I originally claimed -- they actually depend on the data the model is tested on. Furtermore, the probabilistic interpretation should always be conditioned on this data $\mathcal{D}$, e.g. $TPR({\mathcal{D}}) = \Pr(\widehat{y}=1|y=1,\mathcal{D})$. And, in general, $TPR({\mathcal{D}_1}) \neq TPR({\mathcal{D}_2})$ unless $\mathcal{D}_1$ and $\mathcal{D}_2$ are close. Is this correct? $\endgroup$
    – Alexandru Dinu
    Commented Apr 19, 2022 at 21:36
  • $\begingroup$ Disregarding calibration for a moment and just considering $y = 1$ when classifier output $> 0.5$, then the TPR will be the same for both datasets, i.e. it does not matter the actual value over or under the threshold, just the fact that you got a correct classification. Is something wrong with this argument? $\endgroup$
    – Alexandru Dinu
    Commented Apr 20, 2022 at 7:35
  • $\begingroup$ In response to your third comment: When the classifier output is greater than 0.5 we don't know that the actual value is definitely True. If the output of the classifier is 0.6 (and our classifier is well calibrated) then we might always predict True, but the actual value will only be True roughly 60% of the time. $\endgroup$
    – Ryan Volpi
    Commented Apr 20, 2022 at 14:37
  • $\begingroup$ I would say "easy to classify" is close to what I mean by deterministic. $\endgroup$
    – Ryan Volpi
    Commented Apr 20, 2022 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.