# In VAEs, why don't we just use a fixed variance for the z distribution?

I'm practicing with VAEs for generative purpose, and from what I understood we need the latent variable $z$ to approximate a distribution, usually the standard normal $N(0, I)$.

In any example I could find we use $z \sim N(\mu(h;\theta), \Sigma(h;\theta))$ where $\theta$ are the optimization parameters and $h$ is the encoder's output.

It is very apparent to me that $\mu$ should depend from $h$ because that's the way the information gets from the encoder to the latent space, but I cannot understand the choice of having $\Sigma = \Sigma(h;\theta)$.

In all my optimization attempts the realization of $\Sigma$ consistently just gets near $I$ as much as possible, hence I guess I could simply use $\Sigma = I$ or maybe $\Sigma = \Sigma(\theta)$ with no dependency on $h$.

This would reduce the number of parameters improving speed. Why don't we actually do that?

To my knowledge, the output of the encoder $$(\mu(x),\Sigma(x))=f_E(x)$$ is usually the only thing used to parametrize the resulting encoder distribution, i.e. $$q(z|x) = \mathcal{N}(z|\mu(x),\Sigma(x))$$. Now, let me give a conceptual, then a practical reason why for your question.
Theoretically speaking, our encoder distribution is a variational approximation to the true posterior. In other words, the variational posterior is being fit to the true posterior. By removing the $$\Sigma$$, you are weakening this fit. Generally, using a normal approximation is already a pretty weak approximation (that's why techniques like Variational Inference with Normalizing Flows exist), so you don't really want to weaken it even more!
Practically speaking, you often don't actually want $$\Sigma(x)=I$$. It really depends on the situation. Keep in mind that you are optimizing the ELBO, not the true marginal likelihood, which makes it seems like $$\Sigma(x)=I$$ is ideal (since it minimizes $$\mathfrak{D}_\text{KL}[q(z|x)\mid\mid p(z)]$$, where $$p(z)=\mathcal{N}(0,I)$$). However, it would be better to be minimizing $$\mathfrak{D}_\text{KL}[q(z)\mid\mid p(z)]$$ where $$q(z) = \int q(z|x) p_\text{empirical}(x) dx$$, but this is intractable. Clearly this shows that $$q$$ should be adaptive based on the input. Otherwise, the marginal posterior may not match the prior as well.