For a neural net where the output is a Gaussian distribution, the output is usually parameterized as $(\mu=O_1, \sigma^2=e^{O_2})$. That is to say, the neural net will output the mean, and also output the log of the variance $O_2$. This is the natural choice to make all the outputs fit on the real line, and works well when training the neural net.

However, I want to output a bivariate Gaussian distribution (with 2 means $\mu_1$ and $\mu_2$, 2 variances $v_1$ and $v_2$, and a correlation $\rho$). The problem is, there are several natural ways to parameterize the variances and correlation (such that everything is a real number).

To name just 2:

  1. Each parameter gets its own output. \begin{align} v_1 &= e^{O_1} \\ v_2 &= e^{O_2} \\ \rho &= tanh(O_3) \end{align}

  2. Say that \begin{align} X_1 &= Z_1 + Z_c \\ X_2 &= Z_2 + Z_c \end{align} where $Z_i\sim\mathcal{N}(0, e^{O_i})$ are "private information" and $Z_c\sim\mathcal{N}(0, e^{O_3})$ is a "shared" gaussian information. Then we will have that \begin{align} v_1 &= e^{O_1} + e^{O_3} \\ v_2 &= e^{O_2} + e^{O_3} \\ \rho &= \frac{e^{O_3}}{\sqrt{v_1 \cdot v_2}} \end{align}

Which of these ways of parameterizing the output bivariate Gaussian distribution would be best? Or is there a better way? My main criteria would be what works best when training a neural network.

Personally, I think that the second way would be better, since the choice of correlation should be closely tied to the choice of variances, and the "private information"/"shared information" model for a bivariate Gaussian seems very natural.

Finally, I'm satisfied with just the bivariate case, but I may as well ask what one could do in the multivariate case?


1 Answer 1


For the multivariate case, you could try outputting a vector $d$ of diagonal entries and an $n\times k$ matrix $R$, and let $\Sigma = \text{diag}(d) + RR^T$. This would be easy to sample, easy to backpropagate through, and also you can trade off computation cost with expressivity (as $k \rightarrow n$, you have a full covariance matrix).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.