# An identity involving conditional expectations

Since my question did not receive an answer, I try to improve it and make it more explicit.

I would like to understand the reasoning behind this passage ( par. D.5, page 19, https://arxiv.org/pdf/1605.07723.pdf ). Here the passage I would like to understand (in their minds, it should be trivial ):

Here $$x,y$$ are r.v., $$f,\lambda$$ some discrete functions, and $$w$$ parameters.

The conditional independence assumption they are referring to is:

$$y \perp f(x) | \lambda(x)$$

but I was not able to use it to derive the third line from the second.

As far as I understand, in the second line the internal expectation is a conditional expectation over $$y$$ (since $$x$$ does not appear in the function, but only $$\overline{x}$$), where the distribution of $$y$$ is $$p(y|x=\overline{x})$$, whereas in the third line we still can consider it as an expectation over $$y$$ (since $$x$$ does not appear in the function), but under a different conditional distribution $$p(y|\lambda(x)=\lambda(\overline{x}))$$. I am not able at the moment to use the conditional independence assumption to derive the third line from the second. I think we should use that $$p(y|f(x),\lambda(x))=p(y|\lambda(x))$$, so that in some sense the distribution of $$y$$ does not depend on the value of $$f(x)$$, once we condition on $$\lambda(x)$$. This would suggest maybe to write things like $$E_x \rightarrow E_{\lambda} E[...|\lambda]$$ (symbolically), but I did not arrive to any conclusion.

Any help/suggestion ?

PREVIOUS/OLD VERSION OF THE QUESTION:

I try to make the argument abstract, so that it should not be mandatory to read the article in order to understand the question.

As far as I understand we have the setting:

1 $$X, Y$$ are random variables

[2] $$f(X), \lambda(X)$$ are deterministic functions of $$X$$ ( so themselves random variable)

[3] $$Y \perp f(X) | \lambda(X)$$

[4] $$g=g(f(X),Y))$$ is a deterministic function of $$f(X),Y$$. In the article they have $$g(f(X),Y)=log(1+e^{-wf(X)Y})$$, with $$w$$ parameters, but I am not sure the precise functional form matters.

The goal would be to exploit the properties to rewrite $$E[g(f(X),Y)]$$ in a different manner.

First they use something similar to the law of total expectation. Given $$\overline{X},\overline{Y}$$ distributed as $$X,Y$$ but independent w.r.t to them one can first write:

$$E_{X,Y}[g(f(X),Y)]=E_{\overline{X},\overline{Y}}[E_{X,Y}[g(f(\overline{X}),Y)|X=\overline{X}]]$$ [Eq 1]

Than they say:

$$E_{X,Y}[g(f(X),Y)]=E_{\overline{X},\overline{Y}}[E_{X,Y}[g(f(\overline{X}),Y)|\lambda(X)=\lambda(\overline{X})]]$$ [Eq 2]

I guess here they are using the partial independence hypothesis, that says that fixing $$\lambda(X)$$ to some value makes $$Y$$ and $$f(X)$$ independent, but I am not sure how exactly how to apply this. Should it be trivial how to do that ?

My question is how the authors managed to derive [Eq 2] starting from [Eq 1].

UPDATE:

Following some comments in the chat, an idea is to start writing:

$$E_{\overline{X},\overline{Y}}[E_{X,Y}[g(f(\overline{X}),Y)|X=\overline{X}]]=E_{\overline{X},\overline{Y}}[E_{X,Y} [g(f(\overline{X}),Y)|X=\overline{X},\lambda(X)=\lambda(\overline{X})]]$$

this is true because essentialy if $$X=\overline{X}$$ than $$\lambda(X)=\lambda(\overline{X})$$ and we are just adding redundant information. The big step of course is to get rid of the condition $$X=\overline{X}$$, which I have problems in doing it formally using the conditional independence condition [3].

• If the function $\lambda$ is assumed to be invertible, that is, $\lambda^{-1}$ exists, then $\lambda^{-1}(\lambda(X)) = \lambda^{-1}(\lambda(\bar{X}))$ is the same as $X=\bar{X}$. Jun 5, 2021 at 15:01
• Thanks for the comment. It should not be the case that $\lambda$ is invertible, otherwise I agree that we would just be conditioning on the same events. Jun 5, 2021 at 15:03
• The outer expectation would then need to be with respect to $\lambda(\bar{X})$ for the law of total expectation to be correctly applied. I'm not sure what your question is about then, could you clarify? Jun 5, 2021 at 15:06
• @mhdadk Thanks. I updated my post clarifying the question. See also the linked pdf, page 19, in case I reported something in a wrong way. It should be clear what are the formulas that I am trying to understand. Jun 5, 2021 at 15:12
• In the paper, what is the difference between "class $y$" and "label $\lambda(x)$"? Jun 5, 2021 at 15:18

I think I finally have a formal justification.

We start from here

$$E_{\overline X, \overline Y}E_{X,Y}[g(f(\overline X),Y))|\lambda(X)=\lambda(\overline X)]$$

and define for simplicity $$\Lambda=\lambda(X),F=f(X)$$. Than the previous expression reads, if we try to make the expectations more explicit:

$$\int df d\lambda \ p(F=f, \Lambda=\lambda) \int dy \ g(f,y) p(Y=y | \Lambda=\lambda)$$ [1]

Let's use now that because of the conditional independence assumption :

$$p(Y=y | \Lambda=\lambda)=p(Y=y | \Lambda=\lambda, F=f)$$ [2]

and:

$$p(F=f, \Lambda=\lambda)=p(\Lambda=\lambda|F=f)p(F=f)$$ [3]

Inserting 2 and 3 into 1, we can perform the integral over $$\lambda$$ using that:

$$\int d\lambda p(Y=y | \Lambda=\lambda, F=f)p(\Lambda=\lambda|F=f)=p(Y=y|F=f)$$

what remains is:

$$\int df dy g(f,y) p(Y=y|F=f)p(F=f)=\int df dy g(f,y) p(Y=y,F=f)=E[g(f(X),Y)]$$

, which is the desired result.

The law of the unconscious statistician has been used several times.

UPDATE:

We can try to reverse the argument. Whenever we want to compute:

$$E[f(X,Y)]$$

and we know that $$X$$ is independent of $$Y$$ given $$Z$$, than:

$$E[f(X,Y)]=\int dx dz\ p(z,x) E_{Y \sim p(y|z)}[f(x,Y)]$$

. This can be useful whenever we cannot sample the joint $$p(x,y)$$ but can sample $$x$$, the joint $$p(x,z)$$, and we know the distribution $$p(y|z)$$, i.e. we know quite well how the variables $$X$$ and $$Y$$ are related to $$Z$$, but we have less control on the relation between the variables.

For example (this should be the setting in the article) $$X$$ may be a covariate, $$Y$$ an unobserved dependent variable, $$Z$$ a function of $$X$$ that we can measure (an approximation of $$Y$$) and we have a model of $$p(y|z)$$ that we can fit independently.