For the vanilla autoencoder the structure is like this:

It can be treated as a nonlinear extension of PCA, while for the variational autoencoder a mean and a standard deviation is added as a layer for each hidden variable in the middle layer:

For the detailed difference please refer to this answer.

Should VAEs be even used for non-generative tasks?

Yes, you can. The additional KL divergence loss can be just seen a regularization and regularization can reduce the variance(at the risk of increasing bias).

If I were to use both models for embedding images, how would the embedding space differ on a structural level?

For the VAE the values of embedding would just be like samples from the normal distribution while that doesn't hold for the general autoencoder.

Reference:
Intuitively Understanding Variational Autoencoders

For the vanilla autoencoder the structure is like this:

It can be treated as a nonlinear extension of PCA, while for the variational autoencoder a mean and a standard deviation is added as a layer for each hidden variable in the middle layer:

For the detailed difference please refer to this answer.

Should VAEs be even used for non-generative tasks?

Yes, you can. The additional KL divergence(between variational distribution and the prior distribution/normal distribution) loss can be just seen a regularization and regularization can reduce the variance(at the risk of increasing bias).

If I were to use both models for embedding images, how would the embedding space differ on a structural level?

For the VAE the values of embedding would just be like samples from the normal distribution while that doesn't hold for the general autoencoder.

Reference:
Intuitively Understanding Variational Autoencoders

For the vanilla autoencoder the structure is like this:

It can be treated as a nonlinear extension of PCA, while for the variational autoencoder a mean and a standard deviation is added as a layer for each hidden variable in the middle layer:

For the detailed difference please refer to this answer.

Reference:
Intuitively Understanding Variational Autoencoders

For the vanilla autoencoder the structure is like this:

It can be treated as a nonlinear extension of PCA, while for the variational autoencoder a mean and a standard deviation is added as a layer for each hidden variable in the middle layer:

For the detailed difference please refer to this answer.

Should VAEs be even used for non-generative tasks?

Yes, you can. The additional KL divergence loss can be just seen a regularization and regularization can reduce the variance(at the risk of increasing bias).

If I were to use both models for embedding images, how would the embedding space differ on a structural level?

For the VAE the values of embedding would just be like samples from the normal distribution while that doesn't hold for the general autoencoder.

Reference:
Intuitively Understanding Variational Autoencoders