Why is there covariance instead of simply variance in multidimensional normal distributions?

Question

Maybe it is just because I'm only experimenting with 2 dimensional normal distributions, but multi-dimensional normal distributions for me seem like just multiple one dimensional normal distributions.

However, in this Wikipedia link, the illustration just at the top right corner is a two dimensional normal distribution as well, but it defines covariance ($\sum$), instead of variance ($\sigma^2)$, and I don't understand why it's needed.

I tried to reproduce the same diagram, and (seemingly) successfully implemented it just using variance. The result looks like this:

Built with this python code:

import matplotlib.pyplot as plt
import numpy as np

def gaussian(x, mean, variance):
  return 1 / (variance / np.sqrt(2 * np.pi)) * np.exp(-0.5 * (x - mean)**2 / variance**2)

means = [4, 4]
variances = [1, 0.8]

placeholder = np.arange(0, 8, 0.01)

epsilon = np.random.normal(size=(300, 2))
x = np.array(means) + np.exp(0.5 * np.array(variances)) * epsilon

fig = plt.figure()
fig = fig.gca(projection='3d')

fig.plot(placeholder, gaussian(placeholder, means[0], variances[0]), zs=8, zdir='y')
fig.plot(placeholder, gaussian(placeholder, means[1], variances[1]), zs=0, zdir='x')
fig.scatter(x[:,0], x[:,1], zs = 0, zdir='z')

fig.legend()
fig.set_xlim(0, 8)
fig.set_ylim(0, 8)
fig.set_zlim(0, 4)
fig.set_xlabel('X')
fig.set_ylabel('Y')
fig.set_zlabel('Z')

plt.savefig('diagram.png')

I really need to understand this for my long journey to understand VAEs.

Answer 1

In the Wikipedia picture, the scatterplot looks like an ellipse. The major axis of the ellipse isn't parallel to either of the graph axes. You will be unable to replicate this part with zero covariance, because your ellipse's major axis will be parallel to the x axis if your x coordinate has higher variance, or to the y axis if your y coordinate has higher variance.

Answer 2

In multivariate normal distribution, covariance defines how dependent the random variables are. Normally, the PDF will be like

$$f(x,y)=A\exp(a(x-\mu_x)^2+b(y-\mu_y)^2+c(x-\mu_x)(y-\mu_y))$$

But, if you omit the covariances (i.e. assume $0$), the cross term disappears, i.e. $$f(x,y)=A\exp(a(x-\mu_x)^2+b(y-\mu_y)^2)$$

Both forms will have the same marginal PDFs, but they're different multivariate functions. So, the plot in 3D will look different, although their 2D projections onto walls will be the same.