# Empirical Risk Minimization: Rewriting the expected loss using Bayes' rule and the definition of expectation

I am currently studying Transfer Learning by Qiang Yang, Yu Zhang, Wenyuan Dai, and Sinno Jialin Pan. Chapter 2.2 Instance-Based Noninductive Transfer Learning says the following:

As mentioned earlier, in noninductive transfer learning, the source task and the target task are assumed to be the same, and the supports of the input instances across domains are assumed to be the same or very similar, that is, $$\mathscr{X}_s = \mathscr{X}_t$$. The only difference between domains is caused by the marginal distribution of input instances, that is, $$\mathbb{P}_s^X \not= \mathbb{P}_t^X$$. Under this setting, we are given a set of source domain-labeled data $$\mathscr{D}_s = \{ (\mathbf{\mathrm{x}}_{s_i}, y_{s_i} ) \}_{i = 1}^{n_s}$$, and a set of target domain-unlabelled data $$\mathscr{D}_t = \{ ( \mathbf{\mathrm{x}} ) \}_{i = 1}^{n_t}$$. The goal is to learn a précise predictive model for the target domain unseen data.

In the following, we show that, under the assumptions in noninductive transfer learning, one is still able to learn an optimal predictive model for the largest domain even without any target domain-labeled data. Suppose our goal is to learn a predictive model in terms of parameters $$\theta_t$$ for the target domain, based on the learning framework of empirical risk minimization (Vapnik, 1998), the optimal solution of $$\theta_t$$ can be learned by solving the following optimization problem.

$$\theta_t^* = \mathop{\arg \min}\limits_{\theta_t \in \Theta} \mathbb{E}_{(\mathbf{\mathrm{x}}, y) \in \mathbb{P}_t^{X, Y}} [ \mathscr{l}(\mathbf{\mathrm{x}}, y, \theta)], \tag{2.1}$$

where $$\mathscr{l}(\mathbf{x}, y, \theta)$$ is a loss function in terms of the parameters $$\theta_t$$. Since there are no target domain-labeled data, one cannot optimize (2.1) directly. It has been proven by Pan (2014) that, by using the Bayes' rule and the definition of expectation, the optimization (2.1) can be rewritten as follows,

$$\theta_t^* = \mathop{\arg \min}\limits_{\theta_t \in \Theta} \mathbb{E}_{(\mathbf{\mathrm{x}}, y) \sim \mathbb{P}_s^{X, Y}} \left[ \dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} \mathscr{l}(\mathbf{\mathrm{x}}, y, \theta_t) \right], \tag{2.2}$$

which aims to learn the optimal parameter $$\theta_t^*$$ by minimizing the weighted expected risk over source domain-labeled data. In noninductive transfer learning, as $$\mathbb{P}_s^{Y \mid X} = \mathbb{P}_t^{Y \mid X}$$, by decomposing the joint distribution $$\mathbb{P}^{X, Y} = \mathbb{P}^{Y \mid X} \mathbb{P}^X$$, we obtain $$\dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} = \dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x})}}$$. Hence, (2.2) can be further rewritten as
...

It's this part that I don't understand:

It has been proven by Pan (2014) that, by using the Bayes' rule and the definition of expectation, the optimization (2.1) can be rewritten as follows,

$$\theta_t^* = \mathop{\arg \min}\limits_{\theta_t \in \Theta} \mathbb{E}_{(\mathbf{\mathrm{x}}, y) \sim \mathbb{P}_s^{X, Y}} \left[ \dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} \mathscr{l}(\mathbf{\mathrm{x}}, y, \theta_t) \right], \tag{2.2}$$

How does Bayes' rule and the definition of expectation allow us to rewrite it in this way? I've done a lot of research, but I cannot find any other implementations of Empirical risk minimzation that do what the authors have done here.

EDIT: Here's Vapnik's original paper on Principles of Risk Minimization for Learning Theory.

• I'm wonder if there are a couple typos in equation 2.1. Should $E_{(x,y) \in P_t^{X,Y}}[ l(x,y,\theta) ]$ actually be $E_{(x,y) \sim P_t^{X,Y}}[ l(x,y,\theta_t) ]$? Sep 28 '20 at 23:18
• @user20160 Yes, I think so. But I don't think these errors change anything with regards to the Bayes' theorem part? Sep 28 '20 at 23:26
• No, they don't change anything in that regard. Just wanted to make sure before posting the answer below. Sep 29 '20 at 0:35

I'll assume continuous distributions here but, if any variable is discrete, simply replace the corresponding integral with a sum. Recall that the expectation of a function $$f$$ with respect to a continuous distribution $$p(z)$$ is:

$$E_{z \sim p(z)}\big[f(z)\big] = \int_\mathcal{Z} p(z) f(z) dz$$

The objective function in equation 2.1 can therefore be written as an integral:

$$E_{(x,y) \sim P_t^{X,Y}} \big[ \ell(x, y, \theta_t) \big] = \int_\mathcal{X} \int_\mathcal{Y} P_t(x,y) \ell(x,y,\theta_t) dx dy$$

We can multiply by one without changing anything:

$$= \int_\mathcal{X} \int_\mathcal{Y} \frac{P_s(x,y)}{P_s(x,y)} P_t(x,y) \ell(x,y,\theta_t) dx dy$$

Using the definition of expectation again, the above integral can be seen as an expectation w.r.t. $$P_s(x,y)$$:

$$= E_{(x,y) \sim P_s^{X,Y}} \left[ \frac{P_t(x,y)}{P_s(x,y)} \ell(x,y,\theta_t) \right]$$

This is the objective function in equation 2.2. So, the optimization problems in equations 2.1 and 2.2 are equivalent. Note that Bayes' rule wasn't needed here. But, based on the text you quoted, it sounds like they might be about to use it to move to equation 2.3.

• And here I was spending hours trying to understand how Bayes' rule fits into this. Sep 29 '20 at 1:13
• @ThePointer Yeah, assuming Bayes will come in later, the wording "as follows" is a bit ambiguous/misleading there Sep 29 '20 at 3:12
• What are the $\mathcal{X}$ and $\mathcal{Y}$ in $E_{(x,y) \sim P_t^{X,Y}} \big[ \ell(x, y, \theta_t) \big] = \int_\mathcal{X} \int_\mathcal{Y} P_t(x,y) \ell(x,y,\theta_t) dx dy$? Sep 29 '20 at 15:47
• I think this should instead be $\mathscr{X}$ (feature/sample space) and $\mathscr{Y}$ (label space). From earlier in the textbook: "In particular, we denote by $\mathscr{D}_s = \{ ( \mathrm{\mathbf{x}}_{s_i}, y_{s_i} ) \}^{n_s}_{i = 1}$ the source domain labeled data, where $\mathrm{\mathbf{x}}_{s_i} \in \mathscr{X}_s$ is the data instance and $y_{s_i} \in \mathscr{Y}_s$ is the corresponding class label." Sep 29 '20 at 16:00
• @ThePointer Yes, I'm using $\mathcal{X}$ and $\mathcal{Y}$ to refer to feature space and label space. I've treated label space as continuous (as in regression problems). But, the same reasoning holds for classification problems, where label space is discrete. Sep 29 '20 at 17:06