In VAEs, why don't we just use a fixed variance for the z distribution? I'm practicing with VAEs for generative purpose, and from what I understood we need the latent variable $z$ to approximate a distribution, usually the standard normal $N(0, I)$.
In any example I could find we use $z \sim N(\mu(h;\theta), \Sigma(h;\theta))$ where $\theta$ are the optimization parameters and $h$ is the encoder's output.
It is very apparent to me that $\mu$ should depend from $h$ because that's the way the information gets from the encoder to the latent space, but I cannot understand the choice of having $\Sigma = \Sigma(h;\theta)$.
In all my optimization attempts the realization of $\Sigma$ consistently just gets near $I$ as much as possible, hence I guess I could simply use $\Sigma = I$ or maybe $\Sigma = \Sigma(\theta)$ with no dependency on $h$.
This would reduce the number of parameters improving speed. Why don't we actually do that?
 A: To my knowledge, the output of the encoder $ (\mu(x),\Sigma(x))=f_E(x) $ is usually the only thing used to parametrize the resulting encoder distribution, i.e. $q(z|x) = \mathcal{N}(z|\mu(x),\Sigma(x))$. Now, let me give a conceptual, then a practical reason why for your question.
Theoretically speaking, our encoder distribution is a variational approximation to the true posterior. In other words, the variational posterior is being fit to the true posterior. By removing the $\Sigma$, you are weakening this fit. Generally, using a normal approximation is already a pretty weak approximation (that's why techniques like Variational Inference with Normalizing Flows exist), so you don't really want to weaken it even more!
Practically speaking, you often don't actually want $\Sigma(x)=I$. It really depends on the situation. Keep in mind that you are optimizing the ELBO, not the true marginal likelihood, which makes it seems like $\Sigma(x)=I$ is ideal (since it minimizes $ \mathfrak{D}_\text{KL}[q(z|x)\mid\mid p(z)] $, where $p(z)=\mathcal{N}(0,I)$). However, it would be better to be minimizing $ \mathfrak{D}_\text{KL}[q(z)\mid\mid p(z)] $ where $q(z) = \int q(z|x) p_\text{empirical}(x) dx$, but this is intractable. Clearly this shows that $q$ should be adaptive based on the input. Otherwise, the marginal posterior may not match the prior as well.  
