1
$\begingroup$

I assume that I have two normal distributed variables where one depends on the other:

$P(A) \sim N(0,\sigma_a)$

$P(B|A) \sim N(q\cdot A, \sigma_b)$

How can I get the reverse $P(A|B)$ assuming that B now follows a normal $P(B)\sim N(\mu_b, \sigma_b)$ with given mean?

EDIT: The above is something like a structural equation and my model assumes $\overline b=q \cdot \overline a$. Now I imagine that I got samples from this process and I select a subsample such that $B$ is normal distributed with a given mean. I suppose in that case $A$ will have a different mean (and std.dev.?), and I'm interested how to properly show what it is.

$\endgroup$
2
$\begingroup$

It is not appropriate to assume the marginal distribution of $B$, since its form follows logically from your specified distributions for $A \sim \text{N}$ and $B|A \sim \text{N}$. Nevertheless, the form you have assumed is indeed the form that is implied by these distributions. You can find the marginal distribution of interest by writing out the joint density and completing the square to frame it with a single quadratic in the argument $b$. This will give you the parameter values of interest.

$\endgroup$
  • $\begingroup$ I added some clarification to explain what I imagined the setup to be. $\endgroup$ – Gerenuk Dec 4 '18 at 13:21
  • $\begingroup$ Now you are adding new notation that is left undefined, and making further assumptions that are inconsistent with the previous model framework. That does not help. $\endgroup$ – Reinstate Monica Dec 4 '18 at 21:30
1
$\begingroup$

I will use variances instead of standard deviations in notation for random variables.

For any two random variables $A$ and $B$ with respective means $\mu_A, \mu_B$ and respective variances $\sigma_A^2, \sigma_B^2$, what is the "best" estimate of $B$ in terms of $A$? By "best" we mean that if $g(A)$ denotes the estimate of $B$ in terms of $A$, then we seek the function $g(\cdot)$ such that $E[(B-g(A))^2]$, the mean-square error of the estimate, is as small as possible. It is well-known that the minimum mean-square error (MMSE) estimator of $B$ in terms of $A$ is the conditional mean $E[B\mid A]$ of $B$ given $A$. This conditional mean is a random variable and it is a function of $A$, not of $B$ as it seems, that is, $E[B\mid A]$ is the $g(A)$ that we seek. It is also known that $E[g(A)] = E\big[E[B\mid A]\big]$, the mean of this particular function of $A$, happens, by a miracle of modern mathematics, to equal $E[B] = \mu_B$, the mean of $B$.

A more constrained version of the above function optimization problem asks, "What is the linear minimum mean-square error (LMMSE) estimator of $B$ in terms of $A$" where we seek estimators that are constrained to be of the form $\alpha A + \beta$ and we try to find $\alpha$ and $\beta$ such that $E[(B-(\alpha A + \beta))^2]$ is as small as possible. Here the well-known answer is that $$\alpha = \rho\frac{\sigma_B}{\sigma_a}, \quad \beta = \mu_B - \rho\frac{\sigma_B}{\sigma_a}\mu_A = \mu_B - \alpha\mu_A \tag{1}$$t where $\rho$ is the correlation coefficient, and it is also known that this function $\alpha A + \beta$ has mean and variance given by $$E[\alpha A + \beta] = \alpha \mu_A + \beta = \mu_B, \quad \operatorname{var}(\alpha A + \beta) = \sigma_B^2(1-\rho)^2.\tag{2}$$


As pointed out by @Ben, your assumptions that $A$ is a normal random variable $N(0, \sigma_A^2)$ and that the conditional distribution of $B$ given that $A$ has value $a$ is $N(qa,\sigma_b^2)$ (where $q$ and $\sigma_b$ are known constants (not dependent on $a$ in any way)) is equivalent to the assumption that $A$ and $B$ are jointly normal random variables. If you don't like this bald assertion, notice that \begin{align} f_A(a) &\propto \exp\left(-\frac{a^2}{2\sigma_A^2}\right)\\ f_{B\mid A}(b\mid a) &\propto \exp\left(-\frac{(b-qa)^2}{2\sigma_b^2}\right) \end{align} and so the joint density $$f_{A,B}(a,b) = f_{B\mid A}(b\mid a)\cdot f_A(a) \propto \exp(Q(a,b))$$ where $Q(a,b)$ is a quadratic function of $a$ and $b$, that is, after completing the square etc, it will be found that the joint density is a jointly normal (a.k.a. bivariate normal) density.

Now, in the special case when $A$ and $B$ are jointly normal random variables (that is, they have a bivariate normal density), the MMSE estimator $E[B\mid A]$ is a linear function of $A$, and so the MMSE estimator must coincide with the LMMSE estimator, no? For jointly normal random variables, the conditional distribution of $B$ given $A = a$ is a normal distribution with mean and variance given by $$E[B \mid A = a] = \rho\frac{\sigma_B}{\sigma_A}a + \mu_B - \rho\frac{\sigma_B}{\sigma_a}\mu_A, \quad \operatorname{var}(B \mid A = a) = \sigma_B^2(1-\rho^2) \tag{3}$$ where we are told that $\mu_A = 0$ in this instance, and that $$E[B \mid A = a] = qa, \quad \operatorname{var}(B \mid A = a) = \sigma_b^2\tag{4}$$ (Note the subscript is a lower-case $b$). Comparing $(3)$ and $(4)$ we see that it must be that $\mu_B = 0$, and that the known $q$ equals $\rho\frac{\sigma_B}{\sigma_A}$ (remember that we are given the value of $\sigma_A$) while the known $\sigma_b^2$ must equal $\sigma_B^2(1-\rho^2)$. So we get that $$ \rho\sigma_B = q\sigma_A, \sigma_b^2 = \sigma_B^2-\rho^2\sigma_B^2 = \sigma_B^2 - (q\sigma_A)^2\\ \implies \sigma_B^2 = \sigma_b^2 + (q\sigma_A)^2, \quad \rho = \frac{q\sigma_A}{\sqrt{\sigma_b^2 + (q\sigma_A)^2}} \tag{5} $$

So, $A$ and $B$ are zero-mean jointly normal random variables with variances $\sigma_A^2$ and $\sigma_B^2 = \sigma_b^2 + (q\sigma_A)^2$ and correlation coefficient $\rho = \frac{q\sigma_A}{\sqrt{\sigma_b^2 + (q\sigma_A)^2}}$. It follows that the conditional distribution of $A$ given that $B = b$ is a normal distribution with mean and variance $$E[A \mid B = b] = \rho\frac{\sigma_A}{\sigma_B}b, \quad \operatorname{var}(A \mid B = b) = \sigma_A^2(1-\rho^2)$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.