In a normal classification context, the training instances are drawn from a distribution $D$ which is defined over $X \times Y$, where $X$ is the feature space and $Y$ is the label space. When selection bias occurs however, the learning algorithm is provided with training instances from a biased distribution, say $D'$ and the selection can be controlled by some random variable $s$ which can be in $\{0, 1\}$. If $s=1$ the sample is selected, otherwise not. Note, for the training instances that are not selected we don't have any labels.

Currently, most methods attempt to use the unlabeled data to compute some resampling weights. However, I am trying a method in which the correction is applied directly on the distribution, not on the classifier.

To do so I would like to show that the true distribution is actually a composition of two sub-distributions $D_{s=0}$ and $D_{s=1}$.

$D_{s=1}$ is known as it is our biased distribution and $D_{s=0}$ is the one I am trying to approximate with my method.

So far I have written this in the following way:

$$\begin{align} \mathcal{D}=&\{\mathcal{D}_{s=0},\mathcal{D}_{s=1}\}\\ =&\{\{x_i^{s=0},y_i^{s=0}\},\{x_i^{s=1},y_i^{s=1}\}\}\\ \approx&\{\{x_i^{s=0},\hat{y}_i^{s=0}\},\{x_i^{s=1},y_i^{s=1}\}\}\\ \approx&\{\hat{\mathcal{D}}_{s=0},\mathcal{D}_{s=1}\} \end{align}$$

I am pretty sure this is not the correct notation but I cannot find how to properly formulate it in mathematical terms. I would really appreciate any help to re-write this the proper way!


1 Answer 1


I use conditional distributions and write all random variables upper case; "$\sim$" means "distributed according to"; I made $D_0$ of you $D_{s=0}$ etc. Then $(X,Y)\sim D, (X,Y)|S=0 \sim D_0, (X,Y)|S=1 \sim D_1$.
Write $D$ as a mixture: $D=P(S=0)D_0+P(S=1)D_1$. I'm not fully sure of where you want to go and whether this helps you, but you may also be interested in the distribution of $Y|X$, which could be written as $D^{Y|X}$ and $Y|(X,S=i) \sim D_i^{Y|X}, i=0, 1$. If I interpret the location where you put your "estimation hat" correctly, you may have an estimator of $D_0^{Y|X}$. Here you can piece things together in the same mixture way as before: $D^{Y|X}=P(S=0|X)D_0^{Y|X}+P(S=1|X)D_1^{Y|X}$, and approximate it by estimating/putting a hat on $D_0^{Y|X}$. You will still need the distribution of $S$ dependent on $X$, known or estimated.

(This is pretty tentative but agonising a bit about whether this should rather be a response or a comment I went for response anyway.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.