Why are the tied weights in autoencoders transposed and not inverted?

Question

I am currently reading about Autoencoders. From what I understand so far, when we are dealing with a symmetrical autoencoder, a good practice is to tie the weights of the decoder layers to the weights of the encoder layers. With this technique we halve the number of weights in our model, speeding training and limiting the risk of overfitting, since we don't have to learn the weights of the decoder anymore, we just learn the weights of the encoder and set the weights of the decoder accordingly. What I don't understand is the value assigned to the weights of the decoder. Let's say that the autoencoder has a total of $N$ layers (without counting the input layer), so layer $1$ is the first hidden layer, layer $N/2$ is the codding layer, and layer $N$ is the output layer. Let's also say that $W_L$ represents the connection weights of the $L^{th}$ layer.

From what I've read so far, if we tie the weights of the decoder layers to the weights of the encoder layer, then the weights of the decoder layers will be:

$$W_{N - L + 1} = W_L^\intercal$$

where $^\intercal$ denotes the transpose and $L$ ranges from $1, 2,..., N/2$.

This is my confusion. Why are we using the transpose of the encoder layers weights as the decoder layer weights? Since the encoder has the job of projecting our data into a lower dimension, and then the decoder maps this projection back to the original representation of the data, wouldn't it make more sense to have the weights of the decoder be the inverse of the encoder weights, not the transpose? (Or at least the pseudo-inverse)

If we would use the inverse, that weight matrix would try to project the data back to its original space, it would try to undo the initial projection, and therefore it would try to recreate the initial input, which is what the autoencoder is trying to achieve. But we're not using the inverse. We're using the transpose. And this doesn't seem to make any sense to me.

So, why are we using the transpose and not the inverse (or pseudo-inverse)?

Answer 1

I'll give my own somewhat handwavy explanation of why this might work. I'm not an expert; this is just my reasoning. I don't have a source. Though, as with much of deep learning, there may not actually be any particularly strong theoretical reasons here; it just works well in practice.

Consider an autoencoder with a single hidden layer of lower dimensionality than the input, and no activation functions.

Let $W$ be the weight matrix from the input layer to the hidden layer and $V$ be the weight matrix from the hidden layer to the output layer. We want a $V$ that inverts $W$ (so that any input $x$ gets mapped to its hidden representation $Wx$ and then back to $VWx = x$). This is not generally possible, because $W$ maps from a higher to a lower dimensional space and is thus many-to-one. The best we can do for $V$ is the right inverse of $W$, which will map any $x$ in the row space of $W$ back to exactly $x$, and any $x$ not in the row space of $W$ back to the component of $x$ in the row space of $W$ -- the remaining component of $x$ is mapped to $0$ by $W$ and cannot be recovered. (For $W$ to have a right inverse it must have independent rows, but we can safely assume this.) The right inverse of $W$ is $W^T(WW^T)^{-1} = V$.

When the rows of $W$ are orthonormal, $WW^T = I$ and $V = W^T$. So in this case, the transpose of the weights is exactly what we want. What about when the rows of $W$ are not orthonormal? Well, $W$ isn't really a fixed matrix. It is optimised during training. And if $W$ needs to be orthonormal in order for $W^T$ to invert it as well as possible, then it will become so.

Note that requiring $W$ to have orthonormal rows doesn't meaningfully affect the model. Whatever $W$ is, we can form a weight matrix $W'$ with orthonormal rows via row operations on $W$ (this is the Gram-Schmidt process), which we can write as $W' = LW$ ($L$ being the row operation matrix). This just causes the hidden representation to be transformed in a one-to-one manner (from $Wx$ to $L(Wx)$); it contains the same information about the inputs $x$.

To summarise, the optimal choice for $V$ is the right inverse of $W$. This equals $W^T$ only when $W$ has orthonormal rows, but this can become true during the optimisation process.

I think the other answer is correct in that this is done because (right) matrix inverses are expensive to compute. Using the transpose has the same outcome but is much cheaper.

Answer 2

You are right, it would make a lot more sense to me too if they would use the inverse.

There are two reasons why I think they do this:

• I think the calculation of the inverse is more costly than doing a transposed matrix multiplication. As I remember, the differences might not be in pure algorithmic time complexity, rather GPUs are simply better at doing matrix multiplications. (https://scicomp.stackexchange.com/questions/5372/for-which-statistical-methods-are-gpus-faster-than-cpus)
• Transposition is motivated by arguing that AEs are can be looked at as a dimensionality reduction method like PCA when considering linear layers, activation functions. In that case, transposition would be inverse. By training with weight tying, the desire is probably to learn a set of weights where this property is true.