# Why do auto-encoders with 1 hidden layer usually use the output weights/filter as $W=W^T$?

I was trying to understand why for auto-encoders with 1 hidden layer, we usually use the output weights/filter as $W=W^T$. Is there any theoretical justification to use $W=W^T$? Or maybe any way to derive that usage of weights through some optimization problem? Or maybe its just because of practical/empirical evidence? I hope its a little bit of both and its not just arbitrary.

The reason I am asking this question is because of the following section on this CNN autoencoders paper:

The only (very) "hand wavy" justification I have is because it reminds me of PCA (Principal Component Analysis). Let me share it with you. Consider the compressed representation $z^{(i)}$ of some data $x^{(i)}$ given by PCA. Let the space we are projecting to (given by the top k eigenvectors) be denoted by $\{ u_1, ..., u_k \}$. Then the compressed representation of some data point $x^{(i)}$ can be given by:

$$z^{(i)}= \begin{bmatrix} z^{(i)}_1\\ \vdots\\ z^{(i)}_k\\ \end{bmatrix} = \begin{bmatrix} u^Tx^{(i)}_1\\ \vdots\\ u^Tx^{(i)}_k\\ \end{bmatrix} = \begin{bmatrix} u^T\\ \vdots\\ u^T\\ \end{bmatrix} x^{(i)} =Ux^{(i)}$$

therefore we can get the compressed version of all the data point by expressing the above equation as above and stacking it up in a matrix:

$$Z = XU = \begin{bmatrix} {z^{(1)}}^T\\ \vdots\\ {z^{(n)}}^T\\ \end{bmatrix} = \begin{bmatrix} {x^{(1)}}^T\\ \vdots\\ {x^{(n)}}^T\\ \end{bmatrix} \begin{bmatrix} u_1 & \dots & u_k \\ \end{bmatrix}$$

The important part to notice is that the "weights" used to get the compressed/summarized lower dimension representation was obtained by the following matrix multiplication:

$$Z = XU$$

where $Z$ is the latent representaiton, $X$ is the data matrix and $U$ is the top $k$ eigenvalue matrix.

However, to get the reconstructed representation from the summarized representation we do:

$$\hat{X} = ZU^T \approx X$$

where we have that we used as the reconstruction weights $W' = U^T$. This is the kind of "intuitive" reasoning I see to choose such a weights, but it doesn't seem rigorous at all for the 1 hidden layer neural net. Moreover, it doesn't take into account the non-linearity used in the hidden layer either which worries me further. However, I was wondering if somebody knew a better reason to justify such a choosing of weights.

• If the network is linear, then one can easily show that the minimal reconstruction error will be achieved when input weights equal output weights and equal PCA weights. I.e. PCA provides an optimal solution for the linear autoencoder, even if there is no explicit $W=W^\top$ constraint. However, in the presence of nonlinearities I don't see why optimal decoder should necessarily equal optimal encoder. Commented Jul 12, 2015 at 21:11
• Nice insight @amoeba. I wonder if some of the wavelet shrinkage theory could apply here. I.e. for orthogonal transformations, the transpose is the inverse. If the nonlinearity is a ReLU, this is equivalent to a non-negative soft thresholding, which has a closed-form optimal solution via proximal operators. Donoho/Johnstone were able to provide some nice guarantees about thresholding wavelets, and Papyan/Romano/Elad showed the connection to CNNs. I wonder if tied-weight AEs would inherit those properties in a hierarchical way Commented Jun 23, 2021 at 7:08
• Relevant papers: Papyan et al: jmlr.org/papers/volume18/16-505/16-505.pdf Donoho et al: projecteuclid.org/journals/annals-of-statistics/volume-26/… Commented Jun 23, 2021 at 7:09