# $P(w, v \mid x, y)$ is proportional to $P(x \mid w, v) P(x, y \mid v) P(w) P(v)$?

I am currently studying Transfer Learning by Qiang Yang, Yu Zhang, Wenyuan Dai, and Sinno Jialin Pan. Chapter 2.2.1 Discriminatively Distinguish Source and Target Data says the following:

One simple and effective approach to learn the weights is to transform the problem of estimating the marginal probability density ratio to the problem of distinguishing whether an instance is from the source domain or the target domain. This can be formulated as a binary classification problem with data instances from the source domain being labeled as $$1$$ and those from the target domain being labeled as $$0$$.

For example, Zadrozny (2004) proposes a rejection sampling-based method for correcting sample selection bias. The rejection sampling process is defined as follows. A binary random variable $$\delta \in \{1, 0 \}$$, which is called selection variable, is introduced. An instance $$\mathbf{\mathrm{x}}$$ is sampled from the target marginal distribution $$\mathbb{P}_t^X$$ with probability $$P_t(\mathbf{\mathrm{x}})$$, that is, $$P_t(\mathbf{\mathrm{x}}) = P(\mathbf{\mathrm{x}} \mid \delta = 0)$$. Similarly, $$P_s(\mathbf{\mathrm{x}})$$ can be rewritten as $$P_s (\mathbf{\mathrm{x}}) = P (\mathbf{\mathrm{x}} \mid \delta = 1)$$. $$\mathbf{\mathrm{x}}$$ is accepted by the source domain with probability $$P (\delta = 1 \mid \mathbf{\mathrm{x}})$$ or rejected with probability $$P (\delta = 0 \mid \mathbf{\mathrm{x}})$$. In mathematics, with the new variable $$\delta$$, the density ratio for each data instance $$\mathbf{\mathrm{x}}$$ can be formulated as
$$\dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x}})} = \dfrac{P(\delta = 1)}{P(\delta = 0)} \dfrac{P(\delta = 0)}{P(\delta = 1)} \dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x}})}, \tag{2.5}$$ where $$P (\delta)$$ is the prior probability of $$\delta$$ in the union data set of the source domain and the target domain. By using the Bayes, rule and the equivalent forms of $$P_s (\mathbf{\mathrm{x}})$$ and $$P_t (\mathbf{\mathrm{x}})$$ in terms of $$\delta$$, (2.5) can be further reformulated as $$\dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x}})} = \dfrac{P(\delta = 1)}{P(\delta = 0)} \left( \dfrac{1}{P(\delta = 1 \mid \mathbf{\mathrm{x}})} - 1\right).$$ Therefore, the density ratio for each source domain data instance can be estimated as $$\dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x}})} \propto \dfrac{1}{P_{s, t}(\delta = 1 \mid \mathbf{\mathrm{x}})}$$. To compute the probability $$P(\delta = 1 \mid \mathbf{\mathrm{x}})$$, we regard it as a binary classification problem and train a classifier to solve it. After calculating the ratio for each source data instance, a model can be trained by either reweighting each source data instance or performing importance sampling on the source data set.

Following the idea of Zadrozny (2004), Bickel et al. (2007) propose a framework to integrate the density ratio estimation step and the model training step with reweighted source data instances. Let $$\mathbb{P}^X$$ denote the probability density of $$\mathrm{\mathbf{x}}$$ in the union data set of the source domain and the target domain. We can use any classifier to estimate the probability $$P(\delta = 1 \mid x)$$. Suppose the classifier is parameterized by $$\mathbf{v}$$ and the parameters for the final learning model that is trained on the reweighted source domain data are denoted by $$\mathbf{w}$$. All the parameters can be optimized using the maximum a posterior (MAP) approach: $$[ \mathbf{w}, \mathbf{v} ]_{MAP} = \text{arg max}_{\mathbf{w}, \mathbf{v}} P(\mathbf{w}, \mathbf{v} \mid \mathscr{D}_s, \mathscr{D}_t),$$ where $$\mathscr{D}_s$$ and $$\mathscr{D}_t$$ denote the source data set and the target data set, respectively. Note that $$P(\mathbf{w}, \mathbf{v} \mid \mathscr{D}_s, \mathscr{D}_t)$$ is proportional to $$P(\mathscr{D}_s \mid \mathbf{w}, \mathbf{v}) P(\mathscr{D}_s, \mathscr{D}_t \mid \mathbf{v}) P(\mathbf{w}) P(\mathbf{v})$$. Therefore, the MAP solution can be found by maximising $$P(\mathscr{D}_s \mid \mathbf{w}, \mathbf{v}) P(\mathscr{D}_s, \mathscr{D}_t \mid \mathbf{v}) P(\mathbf{w}) P(\mathbf{v})$$.

How did the authors get that $$P(\mathbf{w}, \mathbf{v} \mid \mathscr{D}_s, \mathscr{D}_t)$$ is proportional to $$P(\mathscr{D}_s \mid \mathbf{w}, \mathbf{v}) P(\mathscr{D}_s, \mathscr{D}_t \mid \mathbf{v}) P(\mathbf{w}) P(\mathbf{v})$$?

• Do you have a source for this claim? – gunes Nov 24 '20 at 9:12
• A sketch that may (?) be useful: I would start from Bayes' theorem: $P(A∣B)=\frac{P(A,B)}{P(B)}=\frac{P(B|A)P(A)}{P(B)}$. Then, assuming $P(w,v∣x,y)=P(W \le w, V \le v∣X\le x, Y\le y)$ with $W,V,X,Y$ being random variables: $P(w,v∣x,y) = \frac{P(w,v,x,y)}{P(x,y)} \overset{?}{\propto} P(w,v,x,y)=P(x,y,w|v)P(v)=P(x,y|w,v)P(w|v)P(v)=P(x|y,w,v)P(y|w,v)P(w|v)P(v)$. – PaulG Nov 24 '20 at 13:11
• @PaulG I am about to submit an edit. – The Pointer Nov 24 '20 at 13:12
• @gunes I didn't have time to post the full source, and I thought that it was a simple issue, such as the law of total probability, so the source wouldn't be needed. Since it may be a more complex problem, I've done a full edit. – The Pointer Nov 24 '20 at 13:16

\begin{align} p(w, v|L, T) = p(w|v, L, T)p(v|L, T) (8) \\\\ = p(w|v, L)p(v|L, T) (9) \\\\ ∝ P(L|w, v)P(L, T|v)p(w)p(v) (10) \end{align}
Equation 8 applies the chain rule, 9 exploits that the classifier w is independent of the test data $$T$$, given $$L$$ and the covariate shift v. Equation 10 applies Bayes rule twice and exploits that w is independent of v.