I am currently reading about Autoencoders. From what I understand so far, when we are dealing with a symmetrical autoencoder, a good practice is to tie the weights of the decoder layers to the weights of the encoder layers. With this technique we halve the number of weights in our model, speeding training and limiting the risk of overfitting, since we don't have to learn the weights of the decoder anymore, we just learn the weights of the encoder and set the weights of the decoder accordingly. What I don't understand is the value assigned to the weights of the decoder. Let's say that the autoencoder has a total of $N$ layers (without counting the input layer), so layer $1$ is the first hidden layer, layer $N/2$ is the codding layer, and layer $N$ is the output layer. Let's also say that $W_L$ represents the connection weights of the $L^{th}$ layer.

From what I've read so far, if we tie the weights of the decoder layers to the weights of the encoder layer, then the weights of the decoder layers will be:

$$W_{N - L + 1} = W_L^\intercal$$

where $^\intercal$ denotes the transpose and $L$ ranges from $1, 2,..., N/2$.

This is my confusion. Why are we using the transpose of the encoder layers weights as the decoder layer weights? Since the encoder has the job of projecting our data into a lower dimension, and then the decoder maps this projection back to the original representation of the data, wouldn't it make more sense to have the weights of the decoder be the inverse of the encoder weights, not the transpose? (Or at least the pseudo-inverse)

If we would use the inverse, that weight matrix would try to project the data back to its original space, it would try to undo the initial projection, and therefore it would try to recreate the initial input, which is what the autoencoder is trying to achieve. But we're not using the inverse. We're using the transpose. And this doesn't seem to make any sense to me.

So, why are we using the transpose and not the inverse (or pseudo-inverse)?


You are right, it would make a lot more sense to me too if they would use the inverse.

There are two reasons why I think they do this:

  • I think the calculation of the inverse is more costly than doing a transposed matrix multiplication. As I remember, the differences might not be in pure algorithmic time complexity, rather GPUs are simply better at doing matrix multiplications. (https://scicomp.stackexchange.com/questions/5372/for-which-statistical-methods-are-gpus-faster-than-cpus)
  • Transposition is motivated by arguing that AEs are can be looked at as a dimensionality reduction method like PCA when considering linear layers, activation functions. In that case, transposition would be inverse. By training with weight tying, the desire is probably to learn a set of weights where this property is true.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy