What should I expect if I train a Variational Autoencoder (VAE) with a dataset composed of identical images?

Question

(leaving aside how pointless this might be) Am I right in thinking that, in theory, if I train a VAE with only one image (passing it over and over), the VAE should learn to recreate that image (or a very similar one) when sampling from the latent distribution and passing the it through the decoder?

Or is there something I'm missing that makes the above statement fundamentally wrong?

Thanks.