To me, tying weights in an autoencoder makes sense if we think of the auto encoder as doing PCA. Why in any situation would it make sense to not tie the weights? If we don't tie the weights, would it not try to learn something that is PCA anyway or rather something that might not be as optimal as PCA?

Also, if weights are not tied, it doesn't make sense to me that the auto-encoder is invertible i.e. if the decoder is looking for an inverse operation because it's a mapping between spaces of different dimension which should not invertible.

So, if the weights are not tied then why do we expect the decoder to learn anything meaningful i.e neither PCA nor an inverse operation?

New contributor
tushar is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

You can have asymmetric encoders and decoders. In fact, due to non-linearities, even with tied-weights most AEs are not invertible (you'd need to have inverse activation functions to achieve something like that). Invertible neural networks (INNs) do exist, based on flows, and I recommend you to check them to see what AEs are missing.

Tied-weights make sense only for linear, single layer, AEs (and why bother if you can have PCA already for that scenario?)

  • $\begingroup$ Are you saying that it's doing a non-linear version of PCA, and this non-linear version of PCA is not a "transpose" operation? If we completely discard the matrix after doing PCA in the linear case, can we optimize a function to learn the transpose? A linear decoder should not be able to learn the transpose because the problem is underdetermined. So similarly, it shouldn't work for the non linear case in an architecture like the autoencoder, where the problem of finding this reverse mapping is also underdetermined. So what makes it possible for this underdetermined system to learn? $\endgroup$ – tushar 2 days ago
  • $\begingroup$ @tushar it's not a non-linear PCA (doesn't have any PCA features with the exception of dimensionality reduction). Non linear PCA exists, and is not based on neural networks. The decoder is not the inverse of the encoder, that part is right. And the problem is not underdetermined either, because it doesn't want to transpose the encoder weights, rather reconstruct inputs from compressed codes. $\endgroup$ – Firebug 2 days ago
  • $\begingroup$ "reconstruct inputs from compressed codes." why is this not underdetermined then? It's trying to find a mapping from a smaller space to a larger space $\endgroup$ – tushar 2 days ago
  • $\begingroup$ @tushar it might incur a loss, whose severity depends on the compression, yes $\endgroup$ – Firebug 2 days ago

Your Answer

tushar is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.