Generating probability distribution parameters using a neural network

Question

I am new to neural network therefore, my question might be super basic. I am reading this article:

https://arxiv.org/pdf/1609.09869.pdf

In this article they've made a deep Markov Model in which they use recurrent neural network to generate the probability distribution parameters (mu and sigma). That is they have a string of observables (X) and which they feed to a bi-directional neural network and they use these networks to produce mu and sigma which in turn gives them latent variable (z). I've attached the image as well.

My question is how can you train your network to give distribution parameters? I know how a neural network works generally. So, for each label we have a corresponding input and with that input we get let's say a probability of whether its a cat or a dog in the final softmax layer. But how can we get mu and sigma for distribution parameters?

Answer 1

This is given in the paper

$$\begin{align} \mu_r &= W_{\mu_r}^\text{right} h_t ^\text{right} + b _{\mu_r}^\text{right} \\ \sigma_r^2 &= \operatorname{softplus}(W_{\sigma_r}^\text{right} h_t ^\text{right} + b _{\sigma_r}^\text{right}) \\ \mu_l &= W_{\mu_l}^\text{left} h_t ^\text{left} + b _{\mu_l}^\text{left} \\ \sigma_l^2 &= \operatorname{softplus}(W_{\sigma_l}^\text{left} h_t ^\text{left} + b _{\sigma_l}^\text{left}) \\ \mu_t &= \frac{\mu_r\sigma^2_l + \mu_l\sigma^2_r }{\sigma^2_l + \sigma^2_r} \\ \sigma^2_t &= \frac{\sigma^2_l \sigma^2_r }{\sigma^2_l + \sigma^2_r} \\ \end{align}$$

They are calculated using dense layers, as a function of the outputs of previous layers. What they are assuming in here is that there the parameters are some complicated, possibly non-linear, function of the inputs

$$ y_t \sim \mathcal{N}(\mu(x), \sigma(x)) $$

You can notice that the last two lines are the update rule for Gaussian distribution given the conjugate priors.