I am new to neural network therefore, my question might be super basic. I am reading this article:


In this article they've made a deep Markov Model in which they use recurrent neural network to generate the probability distribution parameters (mu and sigma). That is they have a string of observables (X) and which they feed to a bi-directional neural network and they use these networks to produce mu and sigma which in turn gives them latent variable (z). I've attached the image as well.

enter image description here

My question is how can you train your network to give distribution parameters? I know how a neural network works generally. So, for each label we have a corresponding input and with that input we get let's say a probability of whether its a cat or a dog in the final softmax layer. But how can we get mu and sigma for distribution parameters?


1 Answer 1


This is given in the paper

$$\begin{align} \mu_r &= W_{\mu_r}^\text{right} h_t ^\text{right} + b _{\mu_r}^\text{right} \\ \sigma_r^2 &= \operatorname{softplus}(W_{\sigma_r}^\text{right} h_t ^\text{right} + b _{\sigma_r}^\text{right}) \\ \mu_l &= W_{\mu_l}^\text{left} h_t ^\text{left} + b _{\mu_l}^\text{left} \\ \sigma_l^2 &= \operatorname{softplus}(W_{\sigma_l}^\text{left} h_t ^\text{left} + b _{\sigma_l}^\text{left}) \\ \mu_t &= \frac{\mu_r\sigma^2_l + \mu_l\sigma^2_r }{\sigma^2_l + \sigma^2_r} \\ \sigma^2_t &= \frac{\sigma^2_l \sigma^2_r }{\sigma^2_l + \sigma^2_r} \\ \end{align}$$

They are calculated using dense layers, as a function of the outputs of previous layers. What they are assuming in here is that there the parameters are some complicated, possibly non-linear, function of the inputs

$$ y_t \sim \mathcal{N}(\mu(x), \sigma(x)) $$

You can notice that the last two lines are the update rule for Gaussian distribution given the conjugate priors.

  • $\begingroup$ but how do we use these to get latent variables? $\endgroup$
    – Peter
    Jul 2, 2019 at 10:57
  • $\begingroup$ @Peter get what exactly? The above formulas show how to get them, to estimate the parameters you maximize the likelihood, as described in the paper. The estimated parameters are the latent variables. $\endgroup$
    – Tim
    Jul 2, 2019 at 11:04
  • $\begingroup$ I think I get it now. Thank you @Tim $\endgroup$
    – Peter
    Jul 2, 2019 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.