I'm practicing with VAEs for generative purpose, and from what I understood we need the latent variable $z$ to approximate a distribution, usually the standard normal $N(0, I)$.
In any example I could find we use $z \sim N(\mu(h;\theta), \Sigma(h;\theta))$ where $\theta$ are the optimization parameters and $h$ is the encoder's output.
It is very apparent to me that $\mu$ should depend from $h$ because that's the way the information gets from the encoder to the latent space, but I cannot understand the choice of having $\Sigma = \Sigma(h;\theta)$.
In all my optimization attempts the realization of $\Sigma$ consistently just gets near $I$ as much as possible, hence I guess I could simply use $\Sigma = I$ or maybe $\Sigma = \Sigma(\theta)$ with no dependency on $h$.
This would reduce the number of parameters improving speed. Why don't we actually do that?