Empirical Risk Minimization: Rewriting the expected loss using Bayes' rule and the definition of expectation

Question

I am currently studying Transfer Learning by Qiang Yang, Yu Zhang, Wenyuan Dai, and Sinno Jialin Pan. Chapter 2.2 Instance-Based Noninductive Transfer Learning says the following:

As mentioned earlier, in noninductive transfer learning, the source task and the target task are assumed to be the same, and the supports of the input instances across domains are assumed to be the same or very similar, that is, $\mathscr{X}_s = \mathscr{X}_t$. The only difference between domains is caused by the marginal distribution of input instances, that is, $\mathbb{P}_s^X \not= \mathbb{P}_t^X$. Under this setting, we are given a set of source domain-labeled data $\mathscr{D}_s = \{ (\mathbf{\mathrm{x}}_{s_i}, y_{s_i} ) \}_{i = 1}^{n_s}$, and a set of target domain-unlabelled data $\mathscr{D}_t = \{ ( \mathbf{\mathrm{x}} ) \}_{i = 1}^{n_t}$. The goal is to learn a précise predictive model for the target domain unseen data.

In the following, we show that, under the assumptions in noninductive transfer learning, one is still able to learn an optimal predictive model for the largest domain even without any target domain-labeled data. Suppose our goal is to learn a predictive model in terms of parameters $\theta_t$ for the target domain, based on the learning framework of empirical risk minimization (Vapnik, 1998), the optimal solution of $\theta_t$ can be learned by solving the following optimization problem.

$$\theta_t^* = \mathop{\arg \min}\limits_{\theta_t \in \Theta} \mathbb{E}_{(\mathbf{\mathrm{x}}, y) \in \mathbb{P}_t^{X, Y}} [ \mathscr{l}(\mathbf{\mathrm{x}}, y, \theta)], \tag{2.1}$$

where $\mathscr{l}(\mathbf{x}, y, \theta)$ is a loss function in terms of the parameters $\theta_t$. Since there are no target domain-labeled data, one cannot optimize (2.1) directly. It has been proven by Pan (2014) that, by using the Bayes' rule and the definition of expectation, the optimization (2.1) can be rewritten as follows,

$$\theta_t^* = \mathop{\arg \min}\limits_{\theta_t \in \Theta} \mathbb{E}_{(\mathbf{\mathrm{x}}, y) \sim \mathbb{P}_s^{X, Y}} \left[ \dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} \mathscr{l}(\mathbf{\mathrm{x}}, y, \theta_t) \right], \tag{2.2}$$

which aims to learn the optimal parameter $\theta_t^*$ by minimizing the weighted expected risk over source domain-labeled data. In noninductive transfer learning, as $\mathbb{P}_s^{Y \mid X} = \mathbb{P}_t^{Y \mid X}$, by decomposing the joint distribution $\mathbb{P}^{X, Y} = \mathbb{P}^{Y \mid X} \mathbb{P}^X$, we obtain $\dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} = \dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x})}}$. Hence, (2.2) can be further rewritten as
...

It's this part that I don't understand:

It has been proven by Pan (2014) that, by using the Bayes' rule and the definition of expectation, the optimization (2.1) can be rewritten as follows,

$$\theta_t^* = \mathop{\arg \min}\limits_{\theta_t \in \Theta} \mathbb{E}_{(\mathbf{\mathrm{x}}, y) \sim \mathbb{P}_s^{X, Y}} \left[ \dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} \mathscr{l}(\mathbf{\mathrm{x}}, y, \theta_t) \right], \tag{2.2}$$

How does Bayes' rule and the definition of expectation allow us to rewrite it in this way? I've done a lot of research, but I cannot find any other implementations of Empirical risk minimzation that do what the authors have done here.

EDIT: Here's Vapnik's original paper on Principles of Risk Minimization for Learning Theory.

Answer 1

I'll assume continuous distributions here but, if any variable is discrete, simply replace the corresponding integral with a sum. Recall that the expectation of a function $f$ with respect to a continuous distribution $p(z)$ is:

$$E_{z \sim p(z)}\big[f(z)\big] = \int_\mathcal{Z} p(z) f(z) dz$$

The objective function in equation 2.1 can therefore be written as an integral:

$$E_{(x,y) \sim P_t^{X,Y}} \big[ \ell(x, y, \theta_t) \big] = \int_\mathcal{X} \int_\mathcal{Y} P_t(x,y) \ell(x,y,\theta_t) dx dy$$

We can multiply by one without changing anything:

$$= \int_\mathcal{X} \int_\mathcal{Y} \frac{P_s(x,y)}{P_s(x,y)} P_t(x,y) \ell(x,y,\theta_t) dx dy$$

Using the definition of expectation again, the above integral can be seen as an expectation w.r.t. $P_s(x,y)$:

$$= E_{(x,y) \sim P_s^{X,Y}} \left[ \frac{P_t(x,y)}{P_s(x,y)} \ell(x,y,\theta_t) \right]$$

This is the objective function in equation 2.2. So, the optimization problems in equations 2.1 and 2.2 are equivalent. Note that Bayes' rule wasn't needed here. But, based on the text you quoted, it sounds like they might be about to use it to move to equation 2.3.