3
$\begingroup$

I research covariate shift solutions for ML models. Some papers/books (e.g. "Probabilistic Machine Learning" by Kevin Murphy) claim that one needs different solutions for covariate and label shifts, I claim they are the same.

Definitions:

Covariate shift - P(Y|X) remains the same, P(X) changes between source and target domains.

Label shift - P(X|Y) remains the same, P(Y) changes between source and target domains.

Difference between these two: causality. We speak about covariate shift when X->Y and label shift otherwise.

I claim that if P(Y|X) changes in any point of the feature space X then P(X|Y) changes in the corresponding point of the label space. Therefore, assumption that P(Y|X)=const. implies that P(X|Y)=const. as well in the whole joint P(Y,X).

Am I wrong?

$\endgroup$

1 Answer 1

2
$\begingroup$

No, $P(Y\mid X)$ remaining unchanged over time does not imply that $P(X\mid Y)$ also remains unchanged.

For a small example, take the two discrete distributions:

$$\begin{pmatrix} 0.2 & 0.1 \\ 0.3 & 0.4 \end{pmatrix} ,\qquad \begin{pmatrix} 0.3 & 0.05 \\ 0.45 & 0.2 \end{pmatrix} $$

with $X$ indexing the columns and $Y$ the rows. Then (because the columns are constant multiples from one dist. to the other) $P(Y\mid X)$ is the same for the two, but $P(X\mid Y)$ is not. So this is an instance of covariate shift, but not label shift.

$\endgroup$
3
  • $\begingroup$ Thank you for replying. Let's assume X and Y can take values 1 and 2. The first joint distribution is: P(X=1, Y=1)=0.2 P(X=1, Y=2)=0.3 P(X=2, Y=1)=0.1 P(X=2, Y=2)=0.4 Conditionals of the 1st first distribution: P(Y|X=1) = 1 with P=0.2, 2 with P=0.3 P(Y|X=2) = 1 with P=0.1, 2 with P=0.4 P(X|Y=1) = 1 with P=0.2, 2 with P=0.1 P(X|Y=2) = 1 with P=0.3, 2 with P=0.4 2nd: P(Y|X=1) = 1 with P=0.3, 2 with P=0.45 P(Y|X=2) = 1 with P=0.05, 2 with P=0.2 P(X|Y=1) = 1 with P=0.3, 2 with P=0.05 P(X|Y=2) = 1 with P=0.45, 2 with P=0.2 Aren't they all different? $\endgroup$
    – JeyBial
    Commented Sep 14, 2023 at 7:37
  • $\begingroup$ Sorry for bad formatting of the comment. I couln't make line split (or I don't know how). I hope you will read it and share your thoughts. $\endgroup$
    – JeyBial
    Commented Sep 14, 2023 at 7:45
  • 1
    $\begingroup$ It seems that I got confused on conditionals. The condition is given so all the events given the condition should sum up to 1. $\endgroup$
    – JeyBial
    Commented Sep 14, 2023 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.