# Expected output distribution of a calibrated model on controls-only test set

I have a question related to ML model calibration. I train a well calibrated binary classification model with the two classes being controls and cases. The model outputs a number between 0 and 1 that corresponds to probability of the example being a case. I would like to evaluate on another data set that has only controls. Is there some expected distribution of outputs from M on the "controls only" data set?

Here is my sloppy attempt to formalize my question. Suppose you have a binary classification problem consisting of a set of features and labels $$\{(X_i, y_i)\}$$ where label

$$y_i \in \{0,1\}$$

Suppose we train a well calibrated model

$$M: X \to (0,1)$$

such that if we pick subset of indices $$\{k\}$$ such that

$$P

then as $$\delta \to 0$$

$$E(y_k)=P$$

Suppose I evaluate now on a new data set $$\{(X'_i, y'_i)\}$$, and in the new dataset $$y'_i=0$$ for all $$i$$ (ie we train on a dataset that has both controls and cases, we evaluate on a dataset where there are only controls).

Given that we test on N samples, how many samples do we expect to find in some interval $$P'

Related questions:

Suppose $$(X_i',y_i')$$ is a subset of $$(X_i,y_i)$$ where we specifically chose only control examples, can we calculate the distribution of M on the subsample? Suppose we test on a controls only data set and get a different distribution, is this enough for us to say that the controls in the control-only data set are distributed differently from controls in the training data set (after doing some significance test)?

Edit:

Suppose we also know distribution $$P_{M_X}$$ which is the distribution of outputs of M on the original training set (we also know distribution of $$y_i$$)

Edit2:

Thinking in terms of $$P_{M_X}$$ and $$(X_i',y_i')$$ being a subset of $$(X_i,y_i)$$ is it true that

$$P_{M_X'}dM_X'=\frac{P_{M_X}\left(1-P_{M_X}\right)dM_X}{\int_0^1 P_{M_X}\left(1-P_{M_X}\right)dM_X}$$

the idea here is that $$P_{M_X'}$$ is $$P_{M_X}$$ reweighted by $$1-P_{M_X}$$ (probability of false positive)