Consistent estimator of the expectation of a conditional probability

Question

I'm stuck in a problem where I have distribution distribution $P(\boldsymbol{x})$, from which I know how to sample from (i.i.d.) and two functions of the random variable $\boldsymbol{x}$: $E(\boldsymbol{x})$ and $A(\boldsymbol{x})$ (both uninvertible). The aim is to compute the conditional mean of $A$ given $E$, $\mathbb{E}[A|E]$.

The question is how to do it.

Specifically, I'm writing that

$$P(A|E) = \int P(\boldsymbol{x},A|E)d\boldsymbol{x} = \int P(A|\boldsymbol{x},E)P(\boldsymbol{x}|E)d\boldsymbol{x} = \frac{1}{P(E)}\int P(A|\boldsymbol{x},E)P(E|\boldsymbol{x})P(\boldsymbol{x}) d\boldsymbol{x}$$

I then use $P(A|\boldsymbol{x}) = \delta(A-A(\boldsymbol{x}))$ ($\delta$ is the Dirac-delta) to write

$$\mathbb{E}[A|E] = \int A P(A|E) dA = \frac{1}{P(E)}\int \int A \delta(A-A(\boldsymbol{x}) P(E|\boldsymbol{x})P(\boldsymbol{x}) d\boldsymbol{x} dA $$ Integrating the $\delta(A-A(\boldsymbol{x}))$ out, and using $P(E|\boldsymbol{x}) = \delta(E-E(\boldsymbol{x}))$ leads to

$$\mathbb{E}[A|E] = \frac{1}{P(E)}\int A(\boldsymbol{x}) \delta(E - E(\boldsymbol{x})) P(\boldsymbol{x}) d\boldsymbol{x} \equiv \frac{I(E)}{P(E)}$$

Both $P(E)$ and $I(E)$ are consistently estimated (denoted as $\overline{P}$ and $\overline{I}$) by the calculation of a multidimensional integral using Monte Carlo. However:

1. is $\overline{I}/\overline{P}$ a consistent estimator of $\mathbb{E}[A|E]$?

2. How do I compute the error estimate of this estimator? $\overline{I}$ is not independent of $\overline{P}$...

3. What is the error estimate of this estimator if $\boldsymbol{x}$ are now correlated?

Answer 1

I do not know what functions you are using (I am assuming that they are not stochastic here) but this is the way I would deal with the problem:

1. Instead of defining $P(E|X)$ as a Dirac Delta, define it as a discrete probability distribution such that $$ P(E=E(x)|x)=\begin{cases} 1\;\;\mathrm{if}\;\;E=E(x) \\ 0\;\;\mathrm{otherwise} \end{cases} $$

This makes sense since because $E(x)$ is a function you know the value of $E$ with 100% certainty once you know $x$. As suggested in your comment (A), you can express the above distribution as the Kronecker delta; $$ \delta_{E,E(x)} = P(E=E(x)|x) $$ 2. Using Bayes Theorem... $$ P(X|E) = \frac{P(E=E(x)|x)P(X)}{P(E)} = \delta_{E,E(x)} \cdot \frac{P(X)}{P(E)} $$ The key here is that the support for $P(X|E)$ is a subset of the support for $P(X)$. i.e. Let $x \in \mathcal{X}$ and $x|E \in \mathcal{X_E}$ where $\mathcal{X_E} \subseteq \mathcal{X}$. Note that for this particular problem the Kronecker Delta $\delta_{x \in \mathcal{X_E}}$ is equivalent to $\delta_{E,E(x)}$.*

You can then define

$$ P(E) =\begin{cases} \sum_{x \in \mathcal{X_E}} P(x)\;\;\mathrm{if}\;\;\mathcal{X_E}\;\mathrm{is\;countable}\\ \;\\ \int_{\mathcal{X_E}} P(x)dx\;\;\mathrm{Otherwise} \end{cases} $$

With this notation you can also redefine your conditional expectation, $$ \mathbb{E}[A|E] = \mathbb{E}[A|x \in \mathcal{X_E}] $$

Once you know $E$ you should be able to define the set $\mathcal{X_E}$. In the trivial case that $E(x)$ was invertible, $\mathcal{X_E}$ would only contain a single number. Sampling from $P(X|E)$ can be accomplished with Monte-Carlo, Metropolis-Hastings, or direct calculation depending upon the nature of $\mathcal{X_E}$.

You can then write the conditional expectation as $$ \mathbb {E}[A|x \in \mathcal{X_E}] =\begin{cases} \sum_{x \in \mathcal{X_E}} A(x) \frac{P(X)}{P(E)}\;\; \mathrm{if} \;\; \mathcal{X_E}\;\mathrm{is\;countable} \\ \; \\ \int_{\mathcal{X_E}} A(x) \frac{P(X)}{P(E)}dx \;\; \mathrm{otherwise} \end{cases} $$ and under some regularity conditions this would be a consistent estimator.

To address your concerns in comment (B), it may very well be more computationally efficient to simulate a joint distribution $(A(x_i),E(x_i))$ if you want to estimate the conditional value of $A$ in respect to many $E^*$. In such a setting you could either average $A(x_i)$ over a weighted sample (with larger weights assigned to values closer to $E^*$), or average $A$ over a smaller sub-sample interval around $E^*$. Whether or not this technique is efficient in the statistical sense depends a lot on the functions you are using. I think it would be a consistent estimator....but I am not entirely sure.

I guess part of the problem is that I do not know what functions you are using. If, for example, $E(x) = x^2$ (totally trivial I know...) then it would be easy to implement the technique I discussed above. With very complicated functions I could see how it would be a total pain, especially if you had to implement an MCMC for each individual $E^*$.