1
$\begingroup$

When we apply Bayes' rule in machine learning, we want to compute the posterior probability $P(y|X)$ by multiplying two probability distributions (the observed class-conditional likelihood $P(X|y)$ and the prior probability $P(y)$) and then dividing by the evidence $P(X)$, like so:

$$ P(y|X) = \frac{P(X|y) \times P(y)}{P(X)} $$

Suppose that the likelihood and prior terms are both Gaussian normal. Let's say $P(X|y) = \mathcal{N}(10.0, 4.0)$ and $P(y) = \mathcal{N}(12.0, 4.0)$. So we need to compute the product of these two Gaussian distributions.

Ater watching an exciting video

Screenshot from ML lesson

I've learned that the product of two Gaussian distributions is another Gaussian::

$$ \mathcal{N}(\mu_{1}, \sigma_1^{2}) \times \mathcal{N}(\mu_{2}, \sigma_1^{2}) = \mathcal{N}(\mu_{3}, \sigma_3^{2}) $$ where

\begin{align} \mu_3 &= \frac{\sigma_2^{2} \mu_1 + \sigma_1^{1} \mu_2}{\sigma_1^{2} + \sigma_2^{2}} \\ \sigma_3^{2} &= \frac{1}{\frac{1}{\sigma_1^{2}} + \frac{1}{\sigma_2^{2}} } \end{align}

In this example, $\mathcal{N}(10.0, 4.0) \times \mathcal{N}(12.0, 4.0) = \mathcal{N}(11.0, 2.0)$

However, that last formula is not what I get when I multiply the two Gaussian series 1 and 2. Here's some Python code to demonstrate what I mean.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import math

def normal(x, mu, sigma_sq):
    # Computes Gaussian probability from
    # https://en.wikipedia.org/wiki/Gaussian_function

    normalizer = 1.0 / np.sqrt(2.0 * math.pi * sigma_sq)
    exp_term = - 0.5 * (x - mu)**2 / sigma_sq
    return normalizer * np.exp(exp_term)

x = np.arange(0, 20, 0.2)
y1 = normal(x, 10, 4)
y2 = normal(x, 12, 4)
y3 = normal(x, 11, 2)
y4 = y1 * y2

So:

  • y1 is $\mathcal{N}(\mu_{1}, \sigma_1^{2})$

  • y2 is $\mathcal{N}(\mu_{2}, \sigma_1^{2})$.

  • y3 is $\mathcal{N}(\mu_{1}, \sigma_1^{2}) \times \mathcal{N}(\mu_{2}, \sigma_1^{2}) = \mathcal{N}(\mu_{3}, \sigma_3^{2})$.

  • And y4 is the element-wise product y1 * y2.

Now, when I plot these four series y1 (blue), y2 (yellow), y3 (green), and y4 (red), the results of y3 and y4 are not the same!

plt.figure(figsize=(10, 7))
plt.plot(x, y1, label='y1 = N(10, 4)', color='b')
plt.plot(x, y2, label='y2 = N(12, 4)', color='y')
plt.plot(x, y3, label='y3 = N(10, 4) x N(12, 4) = N(11, 2)', color='g')
plt.plot(x, y4, label='y4 = y1 * y2', color='r')
plt.legend()
plt.grid(True)

Shouldn't y3 and y4 be the same? Why aren't they?

Gaussians

$\endgroup$
9
  • 7
    $\begingroup$ A product of two densities is not the same thing as the density of a product of two random variables, if that's what you're asking. Indeed, the product of densities is not in general a density but the density of a product is. You should be careful not to conflate the two. $\endgroup$
    – Glen_b
    Jul 31 '21 at 2:20
  • 2
    $\begingroup$ Further, the density of a sum of variables is not a sum of densities. I see such confusions from the misuse of language (saying one but meaning the other) frequently $\endgroup$
    – Glen_b
    Jul 31 '21 at 2:29
  • 1
    $\begingroup$ Glen_b basically gave the answer. A bit of mathematical rigour will help with the spinning head. Try defining what you mean by each statement. In particular, try to explain to yourself whether you can find $X,\,y$ from the equation $P(X|y)=\mathcal{N}(10.0,4.0)$ and why. $\endgroup$ Jul 31 '21 at 2:41
  • 1
    $\begingroup$ In that case you are taking a product of a density and a likelihood (which is not a density), but the product of the two is not a density. As I understand the notation, it is not simply a product of two floating point numbers. It might be best to lay out exactly what X and y are in this case for a more detailed explanation. $\endgroup$
    – Glen_b
    Jul 31 '21 at 3:33
  • 1
    $\begingroup$ Specifically you're apparently asserting P(X|y) and P(y) are densities in your post. P(X|y) may not be a likelihood but rather a conditional distribution, depending on exactly what you're doing, but either way the product is only proportional to a density. $\endgroup$
    – Glen_b
    Jul 31 '21 at 3:58
3
$\begingroup$

The problem is that you’re abusing the notation. What do you exactly mean by $\mathcal{N}(\mu, \sigma^2)$ in here? Is it values of random variable, it’s probability density function, or maybe cumulative distribution function?

You can multiply the probability density functions, this would be equal to their joint density if they are independent

$$ p_X(x) p_Y(y) = p_{XY}(x, y) $$

this is probability of observing $x$ and $y$ happening together. If there would be dependence between them, the results won’t hold by definition. It’s not the same as observing some value $z=xy$, notice that $z$ can have many divisors, so there could be multiple values of $x$ and $y$ that can lead to observing it, so obviously you won’t calculate the probability from probabilities of two particular divisors alone.

For example, imagine that you have two discrete random variables that both can take values in $\{1, 2, 3, 4\}$, $P(XY=4)$ is a probability that depends on observing any of the three pairs of $x$ and $y$ values $(1,4)$, $(2,2)$, $(4,2)$, so it would be equal to

$$P(X=1,Y=4) + P(X= 2, Y= 2)P + P(X= 4, Y= 2)$$

Notice that to calculate $P(X\times Y)$ we summed the probabilities, math on the values does not apply to the math on the distributions.

Your Bayes theorem example is about multiplying conditional probability density $p(x|y)$ by marginal probability density $p(y)$ and dividing by marginal probability density $p(x)$ and the result is another conditional probability density $p(y|x)$. It’s not just multiplying two Gaussian pdf’s to get a third one. Your example didn’t work because you didn’t apply Bayes theorem correctly. Obviously

$$p(x|y)p(y) \ne \frac{p(x|y)p(y)}{p(x)}$$

Moreover, the fact that the result is a Gaussian results from this being a special case of using a conjugate prior. If you used as a prior random variable with other distribution than Gaussian, there wouldn’t be a simple, closed-form solution for the posterior distribution.

You may want to read more on function convolutions. Random variables are functions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.