An identity involving conditional expectations

Question

Since my question did not receive an answer, I try to improve it and make it more explicit.

I would like to understand the reasoning behind this passage ( par. D.5, page 19, https://arxiv.org/pdf/1605.07723.pdf ). Here the passage I would like to understand (in their minds, it should be trivial ):

Here $x,y$ are r.v., $f,\lambda$ some discrete functions, and $w$ parameters.

The conditional independence assumption they are referring to is:

$y \perp f(x) | \lambda(x) $

but I was not able to use it to derive the third line from the second.

As far as I understand, in the second line the internal expectation is a conditional expectation over $y$ (since $x$ does not appear in the function, but only $\overline{x}$), where the distribution of $y$ is $p(y|x=\overline{x})$, whereas in the third line we still can consider it as an expectation over $y$ (since $x$ does not appear in the function), but under a different conditional distribution $p(y|\lambda(x)=\lambda(\overline{x}))$. I am not able at the moment to use the conditional independence assumption to derive the third line from the second. I think we should use that $p(y|f(x),\lambda(x))=p(y|\lambda(x))$, so that in some sense the distribution of $y$ does not depend on the value of $f(x)$, once we condition on $\lambda(x)$. This would suggest maybe to write things like $E_x \rightarrow E_{\lambda} E[...|\lambda]$ (symbolically), but I did not arrive to any conclusion.

Any help/suggestion ?

PREVIOUS/OLD VERSION OF THE QUESTION:

I am reading this article https://arxiv.org/pdf/1605.07723.pdf and I am confused about the step in the proof of lemma D.5 at page 19.

I try to make the argument abstract, so that it should not be mandatory to read the article in order to understand the question.

As far as I understand we have the setting:

1 $X, Y$ are random variables

[2] $f(X), \lambda(X)$ are deterministic functions of $X$ ( so themselves random variable)

[3] $Y \perp f(X) | \lambda(X) $

[4] $g=g(f(X),Y))$ is a deterministic function of $f(X),Y$. In the article they have $g(f(X),Y)=log(1+e^{-wf(X)Y})$, with $w$ parameters, but I am not sure the precise functional form matters.

The goal would be to exploit the properties to rewrite $E[g(f(X),Y)]$ in a different manner.

First they use something similar to the law of total expectation. Given $\overline{X},\overline{Y}$ distributed as $X,Y$ but independent w.r.t to them one can first write:

$E_{X,Y}[g(f(X),Y)]=E_{\overline{X},\overline{Y}}[E_{X,Y}[g(f(\overline{X}),Y)|X=\overline{X}]]$ [Eq 1]

Than they say:

$E_{X,Y}[g(f(X),Y)]=E_{\overline{X},\overline{Y}}[E_{X,Y}[g(f(\overline{X}),Y)|\lambda(X)=\lambda(\overline{X})]]$ [Eq 2]

I guess here they are using the partial independence hypothesis, that says that fixing $\lambda(X)$ to some value makes $Y$ and $f(X)$ independent, but I am not sure how exactly how to apply this. Should it be trivial how to do that ?

My question is how the authors managed to derive [Eq 2] starting from [Eq 1].

UPDATE:

Following some comments in the chat, an idea is to start writing:

$E_{\overline{X},\overline{Y}}[E_{X,Y}[g(f(\overline{X}),Y)|X=\overline{X}]]=E_{\overline{X},\overline{Y}}[E_{X,Y} [g(f(\overline{X}),Y)|X=\overline{X},\lambda(X)=\lambda(\overline{X})]]$

this is true because essentialy if $X=\overline{X}$ than $\lambda(X)=\lambda(\overline{X})$ and we are just adding redundant information. The big step of course is to get rid of the condition $X=\overline{X}$, which I have problems in doing it formally using the conditional independence condition [3].

Answer 1

I think I finally have a formal justification.

We start from here

$E_{\overline X, \overline Y}E_{X,Y}[g(f(\overline X),Y))|\lambda(X)=\lambda(\overline X)]$

and define for simplicity $\Lambda=\lambda(X),F=f(X)$. Than the previous expression reads, if we try to make the expectations more explicit:

$\int df d\lambda \ p(F=f, \Lambda=\lambda) \int dy \ g(f,y) p(Y=y | \Lambda=\lambda)$ [1]

Let's use now that because of the conditional independence assumption :

$p(Y=y | \Lambda=\lambda)=p(Y=y | \Lambda=\lambda, F=f)$ [2]

and:

$p(F=f, \Lambda=\lambda)=p(\Lambda=\lambda|F=f)p(F=f)$ [3]

Inserting 2 and 3 into 1, we can perform the integral over $\lambda$ using that:

$\int d\lambda p(Y=y | \Lambda=\lambda, F=f)p(\Lambda=\lambda|F=f)=p(Y=y|F=f)$

what remains is:

$\int df dy g(f,y) p(Y=y|F=f)p(F=f)=\int df dy g(f,y) p(Y=y,F=f)=E[g(f(X),Y)]$

, which is the desired result.

The law of the unconscious statistician has been used several times.

UPDATE:

We can try to reverse the argument. Whenever we want to compute:

$$E[f(X,Y)]$$

and we know that $X$ is independent of $Y$ given $Z$, than:

$$E[f(X,Y)]=\int dx dz\ p(z,x) E_{Y \sim p(y|z)}[f(x,Y)]$$

. This can be useful whenever we cannot sample the joint $p(x,y)$ but can sample $x$, the joint $p(x,z)$, and we know the distribution $p(y|z)$, i.e. we know quite well how the variables $X$ and $Y$ are related to $Z$, but we have less control on the relation between the variables.

For example (this should be the setting in the article) $X$ may be a covariate, $Y$ an unobserved dependent variable, $Z$ a function of $X$ that we can measure (an approximation of $Y$) and we have a model of $p(y|z)$ that we can fit independently.