3
$\begingroup$

I was trying to understand why for auto-encoders with 1 hidden layer, we usually use the output weights/filter as $W=W^T$. Is there any theoretical justification to use $W=W^T$? Or maybe any way to derive that usage of weights through some optimization problem? Or maybe its just because of practical/empirical evidence? I hope its a little bit of both and its not just arbitrary.

The reason I am asking this question is because of the following section on this CNN autoencoders paper:

enter image description here

The only (very) "hand wavy" justification I have is because it reminds me of PCA (Principal Component Analysis). Let me share it with you. Consider the compressed representation $z^{(i)}$ of some data $x^{(i)}$ given by PCA. Let the space we are projecting to (given by the top k eigenvectors) be denoted by $\{ u_1, ..., u_k \}$. Then the compressed representation of some data point $x^{(i)}$ can be given by:

$$ z^{(i)}= \begin{bmatrix} z^{(i)}_1\\ \vdots\\ z^{(i)}_k\\ \end{bmatrix} = \begin{bmatrix} u^Tx^{(i)}_1\\ \vdots\\ u^Tx^{(i)}_k\\ \end{bmatrix} = \begin{bmatrix} u^T\\ \vdots\\ u^T\\ \end{bmatrix} x^{(i)} =Ux^{(i)} $$

therefore we can get the compressed version of all the data point by expressing the above equation as above and stacking it up in a matrix:

$$ Z = XU = \begin{bmatrix} {z^{(1)}}^T\\ \vdots\\ {z^{(n)}}^T\\ \end{bmatrix} = \begin{bmatrix} {x^{(1)}}^T\\ \vdots\\ {x^{(n)}}^T\\ \end{bmatrix} \begin{bmatrix} u_1 & \dots & u_k \\ \end{bmatrix} $$

The important part to notice is that the "weights" used to get the compressed/summarized lower dimension representation was obtained by the following matrix multiplication:

$$ Z = XU $$

where $Z$ is the latent representaiton, $X$ is the data matrix and $U$ is the top $k$ eigenvalue matrix.

However, to get the reconstructed representation from the summarized representation we do:

$$ \hat{X} = ZU^T \approx X $$

where we have that we used as the reconstruction weights $W' = U^T$. This is the kind of "intuitive" reasoning I see to choose such a weights, but it doesn't seem rigorous at all for the 1 hidden layer neural net. Moreover, it doesn't take into account the non-linearity used in the hidden layer either which worries me further. However, I was wondering if somebody knew a better reason to justify such a choosing of weights.

Improve this question
$\endgroup$
3
  • $\begingroup$ If the network is linear, then one can easily show that the minimal reconstruction error will be achieved when input weights equal output weights and equal PCA weights. I.e. PCA provides an optimal solution for the linear autoencoder, even if there is no explicit $W=W^\top$ constraint. However, in the presence of nonlinearities I don't see why optimal decoder should necessarily equal optimal encoder. $\endgroup$
    – amoeba
    Commented Jul 12, 2015 at 21:11
  • $\begingroup$ Nice insight @amoeba. I wonder if some of the wavelet shrinkage theory could apply here. I.e. for orthogonal transformations, the transpose is the inverse. If the nonlinearity is a ReLU, this is equivalent to a non-negative soft thresholding, which has a closed-form optimal solution via proximal operators. Donoho/Johnstone were able to provide some nice guarantees about thresholding wavelets, and Papyan/Romano/Elad showed the connection to CNNs. I wonder if tied-weight AEs would inherit those properties in a hierarchical way $\endgroup$
    – fr_andres
    Commented Jun 23, 2021 at 7:08
  • $\begingroup$ Relevant papers: Papyan et al: jmlr.org/papers/volume18/16-505/16-505.pdf Donoho et al: projecteuclid.org/journals/annals-of-statistics/volume-26/… $\endgroup$
    – fr_andres
    Commented Jun 23, 2021 at 7:09

1 Answer 1

Reset to default
2
$\begingroup$

You may look at the last paragraph of Section 5 Discussion in Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural computation, 23(7), 1661-1674.

The tied weights rise when you differentiate the energy function w.r.t data to do score matching, see Equation 4.6 and 4.6 in the paper.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Welcome to the site, @HockeySun. This is a nice contribution. Can you elaborate on the answer a little bit? At present, much of the answer relies on sources to which people may not have access. A couple more sentences would be great. $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 27, 2016 at 13:45
  • $\begingroup$ Brilliant! Would you also be able to answer this related question of mine? been wondering about it for months and nobody chipped in stats.stackexchange.com/questions/482658/… $\endgroup$
    – fr_andres
    Commented Jun 23, 2021 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.