# Conditional expectation for non-gaussian variables

Let $A$, $B$ be two zero-mean random variables. Let the variance be $\sigma^2_A$, $\sigma^2_B$ and let the correlation be $\sigma_{AB}$. Consider the following expression :-

$$\mathbb{E}\big[A|B=b\big]$$ When $A,B$ are jointly gaussians the above expression is equal to $$\frac{b\sigma_{AB}}{\sigma^2_B}$$

1. Can we upper bound the above expression for a larger class of distributions?

2. Can we say anything about the expression?

Without making additional assumptions, there is little one can say.

The idea behind the examples given here is that in the regression of $A$ against $B$, we may take a tiny bit of the range of $B$ (for which there is low probability $q$) and drastically alter the conditional distribution of $A$ in that range without changing any of the lower bivariate moments very much.

The bivariate Normal distribution, whose density is shown as a contour plot at the left, has been altered within a narrow strip of horizontal ($B$) values by shifting all vertical ($A$) densities upwards. To compensate, (1) the remaining density was shifted slightly downward (thus rebalancing the expectation of $A$) and (2) the remaining density was contracted in the vertical direction (thus reducing the overall variance of $A$ to compensate for the slight increase caused by the initial shift within the strip).

By making the strip sufficiently narrow, we can shift as much probability of $A$ as we want arbitrarily much (up or down) while keeping the compensating effects on the remainder of the distribution arbitrarily small. (Furthermore, by situating the strip near the middle (horizontally), we can also arrange not to disturb the correlation coefficient if we wish.)

The following explicit example takes this idea to an extreme by supposing both $A$ and $B$ have only two possible values each, but there's nothing special about this extreme case.

Assume $\sigma_A$ and $\sigma_B$ are both nonzero. Given any nonzero numbers $x\gt 0$ and $y$ and $0\lt q \lt 1$, define a bivariate distribution for $(A,B)$ as follows:

\eqalign{ \Pr((A,B)=(0,0)) &= q/2 \\ \Pr((A,B)=(x,0)) &= q/2 \\ \Pr((A,B)=(0,y)) &= 1-q. }

This makes $A$ a multiple $x$ of a Bernoulli$(q/2)$ variable and $B$ is a multiple $y$ of a Bernoulli$(1-q)$ variable. Therefore

$$\sigma^2_A = x^2\left(\frac{q}{2}\right)\left(1-\frac{q}{2}\right);\quad \sigma^2_B = y^2 q(1-q).$$

These determine $x^2$ and $y^2$ in terms of $q$ and the given variances:

$$x^2 = \frac{4\sigma^2_A}{q(2-q)};\quad y^2 = \frac{\sigma^2_B}{q(1-q)}.$$

From the definition of conditional expectation

$$\mathbb{E}(A|B=0) = \frac{0(q/2) + x(q/2)}{q/2 + q/2} = \frac{x}{2} = \frac{\sigma_A}{\sqrt{q(2-q)}}.\tag{1}$$

Obviously we can make the right hand side as large as we wish by making $q$ sufficiently small. Rigorously, if $N \gt 0$, pick $q \gt 0$ such that $q \lt \sigma^2_A/(2N^2)$, so that

$$q(2-q) = 1-(1-q)^2 \lt 1-(1-\sigma^2_A/(2N^2))^2 = \frac{\sigma^2_A}{N^2} - \left(\frac{\sigma_A^2}{2N^2}\right)^2 \lt \frac{\sigma^2_A}{N^2},$$

implying (from expression $(1)$) that

$$\mathbb{E}(A|B=0) = \frac{\sigma_A}{\sqrt{q(2-q)}} \gt \frac{\sigma_A}{\sigma_A/N} = N.$$

Thus we may select $q$ to match the intended variances of the distribution of $(A,B)$ while making $\mathbb{E}(A|B=0)$ arbitrarily large.

• Looks like we might need some smoothness guarantees. Opinion? – Vivek Bagaria May 3 '16 at 22:06
• @Vivek they won't help. There are smooth examples arbitrarily close to the examples given here. – whuber May 3 '16 at 22:29