2
$\begingroup$

Maybe it is just because I'm only experimenting with 2 dimensional normal distributions, but multi-dimensional normal distributions for me seem like just multiple one dimensional normal distributions.

However, in this Wikipedia link, the illustration just at the top right corner is a two dimensional normal distribution as well, but it defines covariance ($\sum$), instead of variance ($\sigma^2)$, and I don't understand why it's needed.

I tried to reproduce the same diagram, and (seemingly) successfully implemented it just using variance. The result looks like this: enter image description here

Built with this python code:

import matplotlib.pyplot as plt
import numpy as np

def gaussian(x, mean, variance):
  return 1 / (variance / np.sqrt(2 * np.pi)) * np.exp(-0.5 * (x - mean)**2 / variance**2)

means = [4, 4]
variances = [1, 0.8]

placeholder = np.arange(0, 8, 0.01)

epsilon = np.random.normal(size=(300, 2))
x = np.array(means) + np.exp(0.5 * np.array(variances)) * epsilon

fig = plt.figure()
fig = fig.gca(projection='3d')

fig.plot(placeholder, gaussian(placeholder, means[0], variances[0]), zs=8, zdir='y')
fig.plot(placeholder, gaussian(placeholder, means[1], variances[1]), zs=0, zdir='x')
fig.scatter(x[:,0], x[:,1], zs = 0, zdir='z')

fig.legend()
fig.set_xlim(0, 8)
fig.set_ylim(0, 8)
fig.set_zlim(0, 4)
fig.set_xlabel('X')
fig.set_ylabel('Y')
fig.set_zlabel('Z')

plt.savefig('diagram.png')

I really need to understand this for my long journey to understand VAEs.

$\endgroup$
2
  • $\begingroup$ VAE = variational autoencoder? $\endgroup$ Commented Dec 9, 2020 at 10:39
  • $\begingroup$ Yes, it is variational autoencoder $\endgroup$ Commented Dec 9, 2020 at 11:07

2 Answers 2

2
$\begingroup$

In the Wikipedia picture, the scatterplot looks like an ellipse. The major axis of the ellipse isn't parallel to either of the graph axes. You will be unable to replicate this part with zero covariance, because your ellipse's major axis will be parallel to the x axis if your x coordinate has higher variance, or to the y axis if your y coordinate has higher variance.

$\endgroup$
1
$\begingroup$

In multivariate normal distribution, covariance defines how dependent the random variables are. Normally, the PDF will be like

$$f(x,y)=A\exp(a(x-\mu_x)^2+b(y-\mu_y)^2+c(x-\mu_x)(y-\mu_y))$$

But, if you omit the covariances (i.e. assume $0$), the cross term disappears, i.e. $$f(x,y)=A\exp(a(x-\mu_x)^2+b(y-\mu_y)^2)$$

Both forms will have the same marginal PDFs, but they're different multivariate functions. So, the plot in 3D will look different, although their 2D projections onto walls will be the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.