# Multiple bivariate normal distributions and conditional expectation

The problem I have is one that ideally seems to be suitable by considering it as a multivariate normal pdf problem however this way seems impossible.

Consider three Normal distributions

$X\sim N(\mu_{x},\sigma_{x})$,

$Y\sim N(\mu_{y},\sigma_{y})$,

$Z\sim N(\mu_{z},\sigma_{z})$.

Now consider the fact that we have the covariance (calculated from sample results) of $X$ and $Z$ and the covariance of $Y$ and $Z$ only. i.e. we cannot calculate any correlation coefficient between $X$ and $Y$.

So I know the correlation $\rho_{xz}$ and $\rho_{yz}$ only.

Now I have the condition $X = a$ and $Y = b$.

How can I calculate the expected value of $Z$?

For $X,Z$ we have $E(Z|X=a) = \rho_{xz} \frac{\sigma_{z}}{\sigma_{x}}(a -\mu_{x})+\mu_{z}$.

For $Y,Z$ we have $E(Z|Y=b) = \rho_{xz} \frac{\sigma_{z}}{\sigma_{y}}(b -\mu_{y})+\mu_{z}$.

Call the first expectation $e_{z|x}$, and the second $e_{z|y}$.

If these two values are different (which seems probable) what can I do to estimate $e_{z|x,y}$ (i.e. expected value of Z given values for X and Y)?

My initial thought is simply to weight the two expectations by the correlation coefficients. So $e_{z|x,y} = \frac{|\rho_{xz}|e_{z|x} +|\rho_{yz}|e_{z|y}}{|\rho_{xz}|+|\rho_{yz}|}$,

to get some sort of a mean estimate from the two above estimates. This assumes the correlations are not zero and their sum is not zero. I am not in any way knowledgeable in this sort of thing so doubt I am right.

Please let me know if there is a better, or even a correct, way to do this, remembering that We do not know the covariance of $X$ and $Y$, not even an estimate from samples. So I don't think we can consider this case a multivariate problem of 3 dimensions.

Edit:

I think Glen_b's answer seems pretty conclusive, but something is niggling me. If we can guess the correlation sign (not the value, just the sign) of X and Y (e.g. we guess $\rho_{xy}$ is positive),it seems the above method would have some merit. Is this just wishful thinking?

Secondly, I'm starting to doubt my own doubts now, because thinking some more, if $\rho_{yz}$ is close to one and $\rho_{xz}$ is close to zero, or very small, then taking the weighted average seems inappropriate since the YZ correlation is far superior to the XZ, so weighting the average is in effect diluting our accuracy and our certainty of getting the right value. Is this thinking valid?

• Your question title says Multiple bivariate normal distributions instead of Multivariate normal distributions. Is there an implication that $(X,Y)$, $(X,Z)$ and $(Y,Z)$ enjoy bivariate normal distributions but that $(X,Y,Z)$ are not multivariate (or trivariate) normal distribution? – Dilip Sarwate Jun 16 '13 at 21:14

Since the answer actually depends on the relationship between $X$ and $Y$, the most you can perhaps do is try to bound the result, and the bounds aren't going to be terribly satisfactory.

Here, I generate some random data (Edit: in R) where the true relationship with each of the variables conditional on other variable is a coefficient of 1.0 and a high conditional correlation (near to 1)

 zz=rnorm(30)
x1=rnorm(30)*.1+sqrt(1-.99)*zz
x2=rnorm(30)*.1-sqrt(1-.99)*zz
y1=rnorm(30)*.1+sqrt(1-.99)*zz


First regression setup - x,y positively related --

  lm(z1~x1+y1)

Call:
lm(formula = z1 ~ x1 + y1)

Coefficients:
(Intercept)           x1           y1
5.173        1.356        2.312


second setup: x,y negatively related

 lm(z2~x2+y1)

Call:
lm(formula = z2 ~ x2 + y1)

Coefficients:
(Intercept)           x2           y1
4.9169       1.1761       0.4482


Okay, there's a fair bit of noise in the estimates, especially of the y coefficient.

Now look at the individual regressions. First setup, where x,y were positively related:

> lm(z1~x1)

Call:
lm(formula = z1 ~ x1)

Coefficients:
(Intercept)           x1
5.213        2.465

> lm(z1~y1)

Call:
lm(formula = z1 ~ y1)

Coefficients:
(Intercept)           y1
5.204        3.179


Now second setup where they were negatively related:

> lm(z2~x2)

Call:
lm(formula = z2 ~ x2)

Coefficients:
(Intercept)           x2
4.934        1.033

> lm(z2~y1)

Call:
lm(formula = z2 ~ y1)

Coefficients:
(Intercept)           y1
4.93185     -0.06165


How are you supposed to get something useful out of the individual relationships without any information about the relationships between the variables?

Short answer: basically, you don't. At least not much.

===

If we can guess the correlation sign (not the value, just the sign) of X and Y (e.g. we guess ρxy is positive),it seems the above method would have some merit. Is this just wishful thinking?

You can reduce the size of the uncertainty - by reducing the potential range of the correlation, there's a smaller range of possible regressions on the two variables.

Secondly, I'm starting to doubt my own doubts now, because thinking some more, if ρyz is close to one and ρxz is close to zero, or very small, then taking the weighted average seems inappropriate since the YZ correlation is far superior to the XZ, so weighting the average is in effect diluting our accuracy and our certainty of getting the right value. Is this thinking valid?

If you knew the thing you say you don't know (the relationship between the two linear predictors), you can work out how much extra information is available in the weaker variable, conditional on the other one - but that only adds information if you know the right coefficient to apply (which will be different from the coefficient in the univariate relationship).

• It would be helpful to others, when posting code, to state what language/software is being used. In fact, I think this should be a requirement for ALL postings that use software. – wolfies Jun 16 '13 at 5:37
• @wolfies You're quite right, that's bad form. I was answering two posts at the same time and got muddled about which one mentioned R. Now mentioned in edit – Glen_b Jun 16 '13 at 6:01
• @Glen_b Thanks for replying. So the jist of your argument is, I simply don't have the information required to make any useful estimate? – Solitude Jun 16 '13 at 6:23
• Unless there's something I've missed, yes, unless you know or can assume something about the relationship between the predictors, you can't tell much from the marginal relationships. – Glen_b Jun 16 '13 at 7:52
• @Glenn_b thanks for answering. I have added an edit to my question. I would be grateful if you could answer. Thanks! – Solitude Jun 16 '13 at 8:55

The covariance matrix $\Sigma$ of $X, Y, Z$ must be nonnegative definite, and since we know $7$ of the $9$ entries (and the $2$ unknown entries have the same value $\operatorname{cov}(X,Y) = c$), we have that $\operatorname{det}(\Sigma)$ is a quadratic in $c$. This may (or it may well not) give useful bounds on $c$ that could be used in determining $E[Z\mid X=a,Y=b]$. For example, with $\sigma_x=\sigma_y=\sigma_z =1$, I get that $$\operatorname{det}(\Sigma) = -c^2+2\rho_{z,x}\rho_{z,y}c + (1-\rho_{z,x}^2-\rho_{z,y}^2)$$ and so $\operatorname{det}(\Sigma) \geq 0$ only if $$\rho_{z,x}\rho_{z,y}-\sqrt{(1-\rho_{z,x}^2)(1-\rho_{z,y}^2)} \leq c \leq \rho_{z,x}\rho_{z,y} + \sqrt{(1-\rho_{z,x}^2)(1-\rho_{z,y}^2)}.\tag{1}$$ For $\rho_{z,x}=\rho_{z,y} = \rho$, we get that $\operatorname{det}(\Sigma) \geq 0$ for $c\in \left[2\rho^2-1, 1\right]$ so that the unknown correlation $c$ is positive for $|\rho| > \frac{1}{\sqrt{2}} \approx 0.7071\ldots$ and exceeds $\frac{1}{2}$ for $|\rho| > \frac{\sqrt{3}}{2}$.

More generally, let $A$ denote the interval of values of $c$ that give $\operatorname{det}(\Sigma) \geq 0$ (cf. $(1)$). Now, we have a formula for the conditional mean $$E[Z\mid X=a, Y=b] = g(c; a,b)$$ in terms of the unknown $c$ and can get bounds

$$\min_{c \in A} g(c; a,b) \leq E[Z\mid X=a, Y=b] \leq \max_{c \in A} g(c; a,b).$$

The usefulness of these bounds remains to be determined.