Variational parameters in variational autoencoders

Question

So we have x as the observed variable, and z as the latent variable, denoted by this bayesian network.

And we parameterize x by $\theta$ to get $p_{\theta}(\mathbf{x})$

the posterior is $p_{\theta}(\mathbf{z} \mid \mathbf{x})$

and the variational posterior is $q_{\phi}(\mathbf{z} \mid \mathbf{x})$ "$\phi$ are the variational parameters which we will optimize over to fit the variational posterior to the exact posterior." = (taken from https://storage.googleapis.com/deepmind-media/UCLxDeepMind_2020/L11%20-%20UCLxDeepMind%20DL2020.pdf )

And in the case of the variational autoencoder, the encoder represents the variational posterior; so when we generate data from the encoder, we use $z=\mu+\sigma \epsilon$

Are $\mu$ and $\sigma^2$ the variational parameters($\phi$)?

secondary question: why is $\theta$ used in the posterior formula, because I think it shouldn't parameterized by $\theta$ since it is a function of Z, not X? I see this notation used in many tutorials/lectures/papers on variational autoencoders.

edit: I found the following passage in Auto Encoding Variational Bayes paper:

" C.1 Bernoulli MLP as decoder In this case let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Bernoulli whose probabilities are computed from $\mathrm{z}$ with a fully-connected neural network with a single hidden layer: $$ \begin{aligned} \log p(\mathbf{x} \mid \mathbf{z}) &=\sum_{i=1}^{D} x_{i} \log y_{i}+\left(1-x_{i}\right) \cdot \log \left(1-y_{i}\right) \\ \text { where } \mathbf{y} &=f_{\sigma}\left(\mathbf{W}_{2} \tanh \left(\mathbf{W}_{1} \mathbf{z}+\mathbf{b}_{1}\right)+\mathbf{b}_{2}\right) \end{aligned} $$ where $f_{\sigma}(.)$ is the elementwise sigmoid activation function, and where $\theta=\left\{\mathbf{W}_{1}, \mathbf{W}_{2}, \mathbf{b}_{1}, \mathbf{b}_{2}\right\}$ are the weights and biases of the MLP.

C.2 Gaussian MLP as encoder or decoder In this case let encoder or decoder be a multivariate Gaussian with a diagonal covariance structure: $$ \begin{aligned} \log p(\mathbf{x} \mid \mathbf{z}) &=\log \mathcal{N}\left(\mathbf{x} ; \boldsymbol{\mu}, \boldsymbol{\sigma}^{2} \mathbf{I}\right) \\ \text { where } \boldsymbol{\mu} &=\mathbf{W}_{4} \mathbf{h}+\mathbf{b}_{4} \\ \log \sigma^{2} &=\mathbf{W}_{5} \mathbf{h}+\mathbf{b}_{5} \\ \mathbf{h} &=\tanh \left(\mathbf{W}_{3} \mathbf{z}+\mathbf{b}_{3}\right) \end{aligned} $$ where $\left\{\mathbf{W}_{3}, \mathbf{W}_{4}, \mathbf{W}_{5}, \mathbf{b}_{3}, \mathbf{b}_{4}, \mathbf{b}_{5}\right\}$ are the weights and biases of the MLP and part of $\boldsymbol{\theta}$ when used as decoder. Note that when this network is used as an encoder $q_{\phi}(\mathbf{z} \mid \mathbf{x})$, then $\mathrm{z}$ and $\mathrm{x}$ are swapped, and the weights and biases are variational parameters $\phi$."

Seems like the weights and biases for the encoder are the variational parameters, while the weights and biases for the decoder are the model parameters. No mention of $\mu$ and $\sigma^2$ as variational parameters,which still puzzles me.

Answer 1

Yes $\mu$ and $\sigma$ are the parameters of our new distribution that approximate the true posterior. We optimize over these parameters to fit the variational posterior to the true posterior, that we cannot compute since it intractable. At the end of the optimization $\mu$ and $\sigma$ describe the latent space.

$\theta$ is used in the posterior because we are after the parameters distribution given the data (evidence). The true posterior is the distribution of $\theta$ given $X$. Since we can't optimize for $\theta$, we approximate it using variational inference and find $\phi$ instead.

The difference is between:

$P(Z|X)$ - the true posterior that is described by $\theta$

and

$P(Q|X)$ - the variational posterior that is described by $\phi$