5
$\begingroup$

Since my question did not receive an answer, I try to improve it and make it more explicit.

I would like to understand the reasoning behind this passage ( par. D.5, page 19, https://arxiv.org/pdf/1605.07723.pdf ). Here the passage I would like to understand (in their minds, it should be trivial ):

enter image description here

Here $x,y$ are r.v., $f,\lambda$ some discrete functions, and $w$ parameters.

The conditional independence assumption they are referring to is:

$y \perp f(x) | \lambda(x) $

but I was not able to use it to derive the third line from the second.

As far as I understand, in the second line the internal expectation is a conditional expectation over $y$ (since $x$ does not appear in the function, but only $\overline{x}$), where the distribution of $y$ is $p(y|x=\overline{x})$, whereas in the third line we still can consider it as an expectation over $y$ (since $x$ does not appear in the function), but under a different conditional distribution $p(y|\lambda(x)=\lambda(\overline{x}))$. I am not able at the moment to use the conditional independence assumption to derive the third line from the second. I think we should use that $p(y|f(x),\lambda(x))=p(y|\lambda(x))$, so that in some sense the distribution of $y$ does not depend on the value of $f(x)$, once we condition on $\lambda(x)$. This would suggest maybe to write things like $E_x \rightarrow E_{\lambda} E[...|\lambda]$ (symbolically), but I did not arrive to any conclusion.

Any help/suggestion ?

PREVIOUS/OLD VERSION OF THE QUESTION:

I am reading this article https://arxiv.org/pdf/1605.07723.pdf and I am confused about the step in the proof of lemma D.5 at page 19.

I try to make the argument abstract, so that it should not be mandatory to read the article in order to understand the question.

As far as I understand we have the setting:

1 $X, Y$ are random variables

[2] $f(X), \lambda(X)$ are deterministic functions of $X$ ( so themselves random variable)

[3] $Y \perp f(X) | \lambda(X) $

[4] $g=g(f(X),Y))$ is a deterministic function of $f(X),Y$. In the article they have $g(f(X),Y)=log(1+e^{-wf(X)Y})$, with $w$ parameters, but I am not sure the precise functional form matters.

The goal would be to exploit the properties to rewrite $E[g(f(X),Y)]$ in a different manner.

First they use something similar to the law of total expectation. Given $\overline{X},\overline{Y}$ distributed as $X,Y$ but independent w.r.t to them one can first write:

$E_{X,Y}[g(f(X),Y)]=E_{\overline{X},\overline{Y}}[E_{X,Y}[g(f(\overline{X}),Y)|X=\overline{X}]]$ [Eq 1]

Than they say:

$E_{X,Y}[g(f(X),Y)]=E_{\overline{X},\overline{Y}}[E_{X,Y}[g(f(\overline{X}),Y)|\lambda(X)=\lambda(\overline{X})]]$ [Eq 2]

I guess here they are using the partial independence hypothesis, that says that fixing $\lambda(X)$ to some value makes $Y$ and $f(X)$ independent, but I am not sure how exactly how to apply this. Should it be trivial how to do that ?

My question is how the authors managed to derive [Eq 2] starting from [Eq 1].

UPDATE:

Following some comments in the chat, an idea is to start writing:

$E_{\overline{X},\overline{Y}}[E_{X,Y}[g(f(\overline{X}),Y)|X=\overline{X}]]=E_{\overline{X},\overline{Y}}[E_{X,Y} [g(f(\overline{X}),Y)|X=\overline{X},\lambda(X)=\lambda(\overline{X})]]$

this is true because essentialy if $X=\overline{X}$ than $\lambda(X)=\lambda(\overline{X})$ and we are just adding redundant information. The big step of course is to get rid of the condition $X=\overline{X}$, which I have problems in doing it formally using the conditional independence condition [3].

$\endgroup$
7
  • $\begingroup$ If the function $\lambda$ is assumed to be invertible, that is, $\lambda^{-1}$ exists, then $\lambda^{-1}(\lambda(X)) = \lambda^{-1}(\lambda(\bar{X}))$ is the same as $X=\bar{X}$. $\endgroup$
    – mhdadk
    Commented Jun 5, 2021 at 15:01
  • $\begingroup$ Thanks for the comment. It should not be the case that $\lambda$ is invertible, otherwise I agree that we would just be conditioning on the same events. $\endgroup$
    – Thomas
    Commented Jun 5, 2021 at 15:03
  • $\begingroup$ The outer expectation would then need to be with respect to $\lambda(\bar{X})$ for the law of total expectation to be correctly applied. I'm not sure what your question is about then, could you clarify? $\endgroup$
    – mhdadk
    Commented Jun 5, 2021 at 15:06
  • $\begingroup$ @mhdadk Thanks. I updated my post clarifying the question. See also the linked pdf, page 19, in case I reported something in a wrong way. It should be clear what are the formulas that I am trying to understand. $\endgroup$
    – Thomas
    Commented Jun 5, 2021 at 15:12
  • $\begingroup$ In the paper, what is the difference between "class $y$" and "label $\lambda(x)$"? $\endgroup$
    – mhdadk
    Commented Jun 5, 2021 at 15:18

1 Answer 1

2
$\begingroup$

I think I finally have a formal justification.

We start from here

$E_{\overline X, \overline Y}E_{X,Y}[g(f(\overline X),Y))|\lambda(X)=\lambda(\overline X)]$

and define for simplicity $\Lambda=\lambda(X),F=f(X)$. Than the previous expression reads, if we try to make the expectations more explicit:

$\int df d\lambda \ p(F=f, \Lambda=\lambda) \int dy \ g(f,y) p(Y=y | \Lambda=\lambda)$ [1]

Let's use now that because of the conditional independence assumption :

$p(Y=y | \Lambda=\lambda)=p(Y=y | \Lambda=\lambda, F=f)$ [2]

and:

$p(F=f, \Lambda=\lambda)=p(\Lambda=\lambda|F=f)p(F=f)$ [3]

Inserting 2 and 3 into 1, we can perform the integral over $\lambda$ using that:

$\int d\lambda p(Y=y | \Lambda=\lambda, F=f)p(\Lambda=\lambda|F=f)=p(Y=y|F=f)$

what remains is:

$\int df dy g(f,y) p(Y=y|F=f)p(F=f)=\int df dy g(f,y) p(Y=y,F=f)=E[g(f(X),Y)]$

, which is the desired result.

The law of the unconscious statistician has been used several times.

UPDATE:

We can try to reverse the argument. Whenever we want to compute:

$$E[f(X,Y)]$$

and we know that $X$ is independent of $Y$ given $Z$, than:

$$E[f(X,Y)]=\int dx dz\ p(z,x) E_{Y \sim p(y|z)}[f(x,Y)]$$

. This can be useful whenever we cannot sample the joint $p(x,y)$ but can sample $x$, the joint $p(x,z)$, and we know the distribution $p(y|z)$, i.e. we know quite well how the variables $X$ and $Y$ are related to $Z$, but we have less control on the relation between the variables.

For example (this should be the setting in the article) $X$ may be a covariate, $Y$ an unobserved dependent variable, $Z$ a function of $X$ that we can measure (an approximation of $Y$) and we have a model of $p(y|z)$ that we can fit independently.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.