0
$\begingroup$

I have a question related to ML model calibration. I train a well calibrated binary classification model with the two classes being controls and cases. The model outputs a number between 0 and 1 that corresponds to probability of the example being a case. I would like to evaluate on another data set that has only controls. Is there some expected distribution of outputs from M on the "controls only" data set?

Here is my sloppy attempt to formalize my question. Suppose you have a binary classification problem consisting of a set of features and labels $\{(X_i, y_i)\}$ where label

$y_i \in \{0,1\}$

Suppose we train a well calibrated model

$M: X \to (0,1)$

such that if we pick subset of indices $\{k\}$ such that

$P<M(X_k)<P+\delta$

then as $\delta \to 0$

$E(y_k)=P$

Suppose I evaluate now on a new data set $\{(X'_i, y'_i)\}$, and in the new dataset $y'_i=0$ for all $i$ (ie we train on a dataset that has both controls and cases, we evaluate on a dataset where there are only controls).

Given that we test on N samples, how many samples do we expect to find in some interval $P'<M(X'_k)<P'+\delta'$


Related questions:

Suppose $(X_i',y_i')$ is a subset of $(X_i,y_i)$ where we specifically chose only control examples, can we calculate the distribution of M on the subsample? Suppose we test on a controls only data set and get a different distribution, is this enough for us to say that the controls in the control-only data set are distributed differently from controls in the training data set (after doing some significance test)?


Edit:

Suppose we also know distribution $P_{M_X}$ which is the distribution of outputs of M on the original training set (we also know distribution of $y_i$)

Edit2:

Thinking in terms of $P_{M_X}$ and $(X_i',y_i')$ being a subset of $(X_i,y_i)$ is it true that

$P_{M_X'}dM_X'=\frac{P_{M_X}\left(1-P_{M_X}\right)dM_X}{\int_0^1 P_{M_X}\left(1-P_{M_X}\right)dM_X}$

the idea here is that $P_{M_X'}$ is $P_{M_X}$ reweighted by $1-P_{M_X}$ (probability of false positive)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.