How $\dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} = \dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x})}}$?

Question

Related: Empirical Risk Minimization: Rewriting the expected loss using Bayes' rule and the definition of expectation

I am currently studying Transfer Learning by Qiang Yang, Yu Zhang, Wenyuan Dai, and Sinno Jialin Pan. Chapter 2.2 Instance-Based Noninductive Transfer Learning says the following:

As mentioned earlier, in noninductive transfer learning, the source task and the target task are assumed to be the same, and the supports of the input instances across domains are assumed to be the same or very similar, that is, $\mathscr{X}_s = \mathscr{X}_t$. The only difference between domains is caused by the marginal distribution of input instances, that is, $\mathbb{P}_s^X \not= \mathbb{P}_t^X$. Under this setting, we are given a set of source domain-labeled data $\mathscr{D}_s = \{ (\mathbf{\mathrm{x}}_{s_i}, y_{s_i} ) \}_{i = 1}^{n_s}$, and a set of target domain-unlabelled data $\mathscr{D}_t = \{ ( \mathbf{\mathrm{x}} ) \}_{i = 1}^{n_t}$. The goal is to learn a précise predictive model for the target domain unseen data.

In the following, we show that, under the assumptions in noninductive transfer learning, one is still able to learn an optimal predictive model for the largest domain even without any target domain-labeled data. Suppose our goal is to learn a predictive model in terms of parameters $\theta_t$ for the target domain, based on the learning framework of empirical risk minimization (Vapnik, 1998), the optimal solution of $\theta_t$ can be learned by solving the following optimization problem.

$$\theta_t^* = \mathop{\arg \min}\limits_{\theta_t \in \Theta} \mathbb{E}_{(\mathbf{\mathrm{x}}, y) \in \mathbb{P}_t^{X, Y}} [ \mathscr{l}(\mathbf{\mathrm{x}}, y, \theta)], \tag{2.1}$$

where $\mathscr{l}(\mathbf{x}, y, \theta)$ is a loss function in terms of the parameters $\theta_t$. Since there are no target domain-labeled data, one cannot optimize (2.1) directly. It has been proven by Pan (2014) that, by using the Bayes' rule and the definition of expectation, the optimization (2.1) can be rewritten as follows,

$$\theta_t^* = \mathop{\arg \min}\limits_{\theta_t \in \Theta} \mathbb{E}_{(\mathbf{\mathrm{x}}, y) \sim \mathbb{P}_s^{X, Y}} \left[ \dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} \mathscr{l}(\mathbf{\mathrm{x}}, y, \theta_t) \right], \tag{2.2}$$

which aims to learn the optimal parameter $\theta_t^*$ by minimizing the weighted expected risk over source domain-labeled data. In noninductive transfer learning, as $\mathbb{P}_s^{Y \mid X} = \mathbb{P}_t^{Y \mid X}$, by decomposing the joint distribution $\mathbb{P}^{X, Y} = \mathbb{P}^{Y \mid X} \mathbb{P}^X$, we obtain $\dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} = \dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x})}}$. Hence, (2.2) can be further rewritten as

$$\theta_t^* = \mathop{\arg \min}\limits_{\theta_t \in \Theta} \mathbb{E}_{(\mathbf{\mathrm{x}}, y) \sim \mathbb{P}_s^{X, Y}} \left[ \dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x}})} \mathscr{l}(\mathbf{\mathrm{x}}, y, \theta_t) \right], \tag{2.3}$$ where a weight of a source domain instance $\mathrm{\mathbf{x}}$ is defined as the ratio of marginal distributions of input instances between the target domain and the source domain at the data point $\mathrm{\mathbf{x}}$.

The joint distribution decomposition $\mathbb{P}^{X, Y} = \mathbb{P}^{Y \mid X} \mathbb{P}^X$ is the chain rule for random variables.

I don't understand how the authors concluded that $\dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} = \dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x})}}$. And nor do I see how Bayes' theorem is used here. So am I missing something here, or is there an error and/or omission?

Answer 1

The main idea behind this is that the conditional distribution $\mathbb P^{Y|X}$ is the same in $s$ and in $t$, that is, $\mathbb P^{Y|X}_s=\mathbb P^{Y|X}_t$. The rationale for this is that we have labeled data from $s$, allowing us to find $\mathbb P^{X,Y}_s$, and unlabeled data from $t$, allowing us to find $\mathbb P^{X}_t$ only. That means the only information regarding the relation between $X$ and $Y$ comes from the domain $s$, and therefore the only way to infer anything about $\mathbb P^{X,Y}_t$ is assuming that $\mathbb P^{Y|X}_s=\mathbb P^{Y|X}_t$. This is an assumption of noninductive transfer learning, which the author wrote as:

The only difference between domains is caused by the marginal distribution of input instances, that is, $\mathbb{P}^X_s \neq \mathbb {P}^X_t$.

Using this equality and the chain rule, we have:

$$\frac{P_t(x,y)}{P_s(x,y)}=\frac{P_t(x)P_t(y|x)}{P_s(x)P_s(y|x)}=\frac{P_t(x)}{P_s(x)}$$

Since $\mathbb P_t(y|x)$ and $\mathbb P_s(y|x)$ cancel out. We did not use the Bayes theorem here, only the chain rule that you mentioned. The reason is that we want to express the joint probability in terms of the marginal $\mathbb P^X$ (which we estimate in both domains) and the conditional distribution $\mathbb P^{Y|X}$ (which we assume equal in both domains, so it cancels out).