I have a question related to ML model calibration. I train a well calibrated binary classification model with the two classes being controls and cases. The model outputs a number between 0 and 1 that corresponds to probability of the example being a case. I would like to evaluate on another data set that has only controls. Is there some expected distribution of outputs from M on the "controls only" data set?

Here is my sloppy attempt to formalize my question. Suppose you have a binary classification problem consisting of a set of features and labels $\{(X_i, y_i)\}$ where label

$y_i \in \{0,1\}$

Suppose we train a well calibrated model

$M: X \to (0,1)$

such that if we pick subset of indices $\{k\}$ such that


then as $\delta \to 0$


Suppose I evaluate now on a new data set $\{(X'_i, y'_i)\}$, and in the new dataset $y'_i=0$ for all $i$ (ie we train on a dataset that has both controls and cases, we evaluate on a dataset where there are only controls).

Given that we test on N samples, how many samples do we expect to find in some interval $P'<M(X'_k)<P'+\delta'$

Related questions:

Suppose $(X_i',y_i')$ is a subset of $(X_i,y_i)$ where we specifically chose only control examples, can we calculate the distribution of M on the subsample? Suppose we test on a controls only data set and get a different distribution, is this enough for us to say that the controls in the control-only data set are distributed differently from controls in the training data set (after doing some significance test)?


Suppose we also know distribution $P_{M_X}$ which is the distribution of outputs of M on the original training set (we also know distribution of $y_i$)


Thinking in terms of $P_{M_X}$ and $(X_i',y_i')$ being a subset of $(X_i,y_i)$ is it true that

$P_{M_X'}dM_X'=\frac{P_{M_X}\left(1-P_{M_X}\right)dM_X}{\int_0^1 P_{M_X}\left(1-P_{M_X}\right)dM_X}$

the idea here is that $P_{M_X'}$ is $P_{M_X}$ reweighted by $1-P_{M_X}$ (probability of false positive)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.