What similarity do VAE encodings have? Say I am training a VAE on MNIST digit data and it learns to reconstruct the digit images. What will the low-dimensional encodings 'z' have in common between shared classes? Will the distances between digit '5' image encodings be smaller compared to the rest? E.g.
|z(image 1 of digit 5) - z(image 2 of digit 5)| < |z(image1 of digit 5) - z(image1 of digit 8)| ?
Or is it the cosine similarity that becomes the smallest in between the class encodings? In other words, given all the encodings for digits 0-9, what similarity measure to use on a new image 'x' to see what class it most likely belongs to?
 A: According to the VAE paper https://arxiv.org/pdf/1312.6114.pdf, for any input $\mathrm{x}^i$, the encoder outputs a mean $\mu^{(i)}$ and a variance $\sigma^{2(i)}$. Then the decoder samples $\epsilon^{(i)}$ from a normal $\mathcal{N}(0,1)$ distribution and applies the reparametrization trick to generate $\mathrm{z}^{(i)}= \mu^{(i)} + \epsilon^{(i)} \sigma^{2(i)}$ so that
$$
\textrm{q}_{\phi} \left(\mathrm{z} | \mathrm{x}^{(i)}\right)= \mathcal{N} \left( z ; \mu^{(i)}, \sigma^{2(i)} I \right)
$$
Now, imagine that the VAE is trained. Denote $z$ for image $i$ of digit $k$ by $z_i^k$. We have $$z_i^k \sim \mathcal{N}(\mu_i^k, {\sigma^2}_i^{k}).$$ Hence, 
$$z_1^5 - z_2^5 \sim \mathcal{N}(\mu_1^5 - \mu_2^5,  {\sigma^2}_1^{5} + {\sigma^2}_2^{5})$$
and,
$$z_1^5 - z_1^8 \sim \mathcal{N}(\mu_1^5 - \mu_1^8,  {\sigma^2}_1^{5} + {\sigma^2}_1^{8})$$
Since $\mu_2^5$ will be closer to $\mu_1^5$ than $\mu_1^8$, $\mu_1^5 - \mu_2^5$ will be closer to 0 than $\mu_1^5 - \mu_1^8$, the probability of the event that 
$|z_1^5 - z_2^5| < |z_1^5 - z_1^8|$ will be very high. 
