# How to parameterize a bivariate Normal distribution output for a neural network?

For a neural net where the output is a Gaussian distribution, the output is usually parameterized as $$(\mu=O_1, \sigma^2=e^{O_2})$$. That is to say, the neural net will output the mean, and also output the log of the variance $$O_2$$. This is the natural choice to make all the outputs fit on the real line, and works well when training the neural net.

However, I want to output a bivariate Gaussian distribution (with 2 means $$\mu_1$$ and $$\mu_2$$, 2 variances $$v_1$$ and $$v_2$$, and a correlation $$\rho$$). The problem is, there are several natural ways to parameterize the variances and correlation (such that everything is a real number).

To name just 2:

1. Each parameter gets its own output. \begin{align} v_1 &= e^{O_1} \\ v_2 &= e^{O_2} \\ \rho &= tanh(O_3) \end{align}

2. Say that \begin{align} X_1 &= Z_1 + Z_c \\ X_2 &= Z_2 + Z_c \end{align} where $$Z_i\sim\mathcal{N}(0, e^{O_i})$$ are "private information" and $$Z_c\sim\mathcal{N}(0, e^{O_3})$$ is a "shared" gaussian information. Then we will have that \begin{align} v_1 &= e^{O_1} + e^{O_3} \\ v_2 &= e^{O_2} + e^{O_3} \\ \rho &= \frac{e^{O_3}}{\sqrt{v_1 \cdot v_2}} \end{align}

Which of these ways of parameterizing the output bivariate Gaussian distribution would be best? Or is there a better way? My main criteria would be what works best when training a neural network.

Personally, I think that the second way would be better, since the choice of correlation should be closely tied to the choice of variances, and the "private information"/"shared information" model for a bivariate Gaussian seems very natural.

Finally, I'm satisfied with just the bivariate case, but I may as well ask what one could do in the multivariate case?

For the multivariate case, you could try outputting a vector $$d$$ of diagonal entries and an $$n\times k$$ matrix $$R$$, and let $$\Sigma = \text{diag}(d) + RR^T$$. This would be easy to sample, easy to backpropagate through, and also you can trade off computation cost with expressivity (as $$k \rightarrow n$$, you have a full covariance matrix).