Can the dimension of the latent space in VAEs, be larger than the dimension of the data?

Question

I am experimenting with VAEs. There, there is a parameter that you need pass when you create the NN, which is the dimension of the latent space.

In the typical MNIST example we have the following data:

# Load digits data 
(X_train, y_train), (X_test, y_test) = keras.datasets.mnist.load_data()

# Print shapes
print("Shape of X_train: ", X_train.shape)
print("Shape of y_train: ", y_train.shape)
print("Shape of X_test: ", X_test.shape)
print("Shape of y_test: ", y_test.shape)

Shape of X_train:  (60000, 28, 28)
Shape of y_train:  (60000,)
Shape of X_test:  (10000, 28, 28)
Shape of y_test:  (10000,)

which after reshaping become:

# Reshape input data
X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)

# Print shapes
print("New shape of X_train: ", X_train.shape)
print("New shape of X_test: ", X_test.shape)

New shape of X_train:  (60000, 784)
New shape of X_test:  (10000, 784)

So after reshaping, we end up with 60k observations with 784 features/dimensions for the training data and 10k observations with 784 features/dimensions for the test data.

In all the examples that I show, they chose a dimension of the latent space which is smaller than 784.

My questions are:

1. what does it mean if you specify the dimension of the latent space, to be larger than 784, lets say 1000 ?

2. Also how does the dimension of the latent space affect the results ? Why (in this MNIST example) would someone choose 100 or 200 instead of 300 ?

Answer 1

As with many neural network questions, the answers are ultimately kind of unsatisfying:

1. It would mean that the dimension of the latent space is larger than 784. Conceptually this doesn't really make sense, since part of the motivation of the VAE is to compress the data into a simpler representation, but there is no mathematical or architectural reason why you couldn't do it.

2. It depends exactly what results you care about. If you care a lot about reconstruction of the data, probably the results will be better with a larger latent space. If instead you care a lot about compressing the data along meaningful dimensions, then probably they will be better with a smaller latent space. But ultimately there is not really any principled way for predicting ahead of time what size latent space will be best for a given dataset, and you either have to just look at what people have done in the past or just try a lot of values for yourself and see what works best.

Answer 2

A guiding principle of autoencoders is that they must obtain a bottleneck.

If they don't, they may simply learn the identity function (i.e. to perfectly replicate the input, but without building any meaningful internal representation of it), which is useless for sampling in the case of VAEs.

I think a practical example of this behaviour is shown in this question.

The dimension of the latent space does affect results, and finding the right dimension brings the tradeoff that Simon stated.