0
$\begingroup$

I'm currently reading the book Probabilistic Machine Learning: An Introduction by Kevin P. Murphy and I'm stuck on the derivation of a formula in section 3.3.4 (Example: inferring an unknown vector)

Given $N$ noisy but independent measurements $\boldsymbol{y}_{n}$ of a quantity $\mathbf{z}\in\mathbb{R}^D$ with $\boldsymbol{y}_{n} \sim \mathcal{N}\left(\boldsymbol{z}, \boldsymbol{\Sigma}_{y}\right)$ the following equality apparently holds:

$$\prod_{n=1}^{N} \mathcal{N}\left(\boldsymbol{y}_{n} \mid \boldsymbol{z}, \boldsymbol{\Sigma}_{y}\right)=\mathcal{N}\left(\overline{\boldsymbol{y}} \mid \boldsymbol{z}, \frac{1}{N} \boldsymbol{\Sigma}_{y}\right)$$

I tried to algebraically verify the equation, which failed, then I coded a simulation in Python in order to empirically verify the above equation. It turns out however that the simulation doesn't result in both sides of the equation being equal. I believe that my simulation is just wrong or ran into numerical issues, but I would be glad if someone could enlighten me, especially by deriving the above formula. See also my code here:

#%% 3.3.4 Example: Inferring an unknown vector.
import numpy as np
from scipy.stats import multivariate_normal, norm

Sigma_y = np.array([[1,0],
                    [0,1]])

Z = np.array([1,1])

Y = np.array([[1,1],[1,1]])

py = lambda y: multivariate_normal.pdf(y, Z, Sigma_y)

N_prod = np.prod([py(y) for y in y])
N_avg = multivariate_normal.pdf(np.mean(Y,axis=0), Z, 1/len(Y)*Sigma_y)
print(f"N_prod: {N_prod}\nN_avg: {N_avg}")
$\endgroup$
7
  • $\begingroup$ The "units" don't look right in that formula. Consider the expectation: the expectation of a product is the product of expectations (en.wikipedia.org/wiki/…). Then scaling $\boldsymbol{z}$ by $k$ would scale the left side by $k^N$ but the right side only by $k$, no? $\endgroup$
    – VRehnberg
    Commented Jul 2, 2022 at 9:40
  • 3
    $\begingroup$ @VRehnberg: the formula need be interpreted as a product of densities, not of random variables. $\endgroup$
    – Xi'an
    Commented Jul 2, 2022 at 9:48
  • $\begingroup$ The equality holds only when $N=1.$ If you take logarithms, the work needed to write a correct equation amounts to the very simple (ordinary) algebra of manipulating a sum of quadratic forms in the variables to show it's a quadratic form in the sum of the variables. $\endgroup$
    – whuber
    Commented Jul 2, 2022 at 14:51
  • $\begingroup$ @whuber The author however claims that it holds for $N > 1$ also. $\endgroup$ Commented Jul 2, 2022 at 15:27
  • $\begingroup$ Then it's either a typo or mistake on the author's part or a misinterpretation on your part. This appears to be an effort to state that the mean of $n$ iid Normal variates follows a Normal distribution with the same mean and $1/n$ times the original variance. That's true, but (interpreting "$\mathcal N$" as the pdf, as you do) the equation is wrong. Write it out for $n=2,$ $D=1,$ $z=0,$ and $\Sigma=(1).$ $\endgroup$
    – whuber
    Commented Jul 2, 2022 at 16:34

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.