Original Question 2
When we take the derivative $\color{red}{\frac{\partial Z}{\partial \mu}}$ or $\color{red}{\frac{\partial Z}{\partial \sigma}}$, we know that
$$
Z=\mu + \sigma * \epsilon \ : \epsilon=N(0,1)
$$
do we sample from $\epsilon$ again to calculate $Z$?
Edited QuestionQuestion 2
I realized I did not do a good job explaining this questions. So, I would like to give it another try. I will try to walk step by step through the sampling of the latent space $(Z)$ and the backprop symbolically.
and easily update the weight with gradient descent. Very straight forward. Note that we have a single value of each partial derivative i.e.: $\frac{\partial HA_1}{\partial H_1}$ - this is an important distinction.
Option 1
So does this make more senseOption 2
We keep the total error formula the same as in the regular neural network except now we have to index because we are going to end up with $n$ of them:
$$
E_i = \frac{1}{m} \sum_{j=1}^{m} e_j
$$
and do backprop after each sample of the latent spaze $Z$ but do not update the weights yet:
$$
\frac{\partial E_i}{\partial w_{16}} = \frac{\partial (\frac{1}{m} \sum_{j=1}^{m} e_j)}{\partial w_{16}}
$$
where i.e.: now we only have one $z$-derivative in the chain unlike $n$ in Option 1
$$
...\frac{\partial Z}{\partial \mu} + ...
$$
and finally update the weights by averaging the gradient:
$$
w_{16}^{k+1} = w_{16}^{k} - \frac{\eta}{n} \sum_{i=1}^{n} \frac{\partial E_i}{\partial w_{16}}
$$
So in Question 2 - is itOption 1 or Option 2 correct? Am I missing anything.?
Thank you so much!
