5
$\begingroup$

To me, tying weights in an autoencoder makes sense if we think of the auto encoder as doing PCA. Why in any situation would it make sense to not tie the weights? If we don't tie the weights, would it not try to learn something that is PCA anyway or rather something that might not be as optimal as PCA?

Also, if weights are not tied, it doesn't make sense to me that the auto-encoder is invertible i.e. if the decoder is looking for an inverse operation because it's a mapping between spaces of different dimension which should not invertible.

So, if the weights are not tied then why do we expect the decoder to learn anything meaningful i.e neither PCA nor an inverse operation?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Please don't cross-post. meta.stackexchange.com/questions/64068/… $\endgroup$
    – Sycorax ♦
    Commented Feb 24, 2021 at 2:14
  • $\begingroup$ Thanks, I'm unable to delete the other question. I'll keep in mind for next time. $\endgroup$
    – tushar
    Commented Feb 24, 2021 at 9:38

1 Answer 1

Reset to default
3
$\begingroup$

You can have asymmetric encoders and decoders. In fact, due to non-linearities, even with tied-weights most AEs are not invertible (you'd need to have inverse activation functions to achieve something like that). Invertible neural networks (INNs) do exist, sharing a lot with normalizing flows, and I recommend you to check them to see what AEs are missing.

Tied-weights make sense only for linear, single layer, AEs (and why bother if you can have PCA already for that scenario?)

Improve this answer
$\endgroup$
4
  • $\begingroup$ Are you saying that it's doing a non-linear version of PCA, and this non-linear version of PCA is not a "transpose" operation? If we completely discard the matrix after doing PCA in the linear case, can we optimize a function to learn the transpose? A linear decoder should not be able to learn the transpose because the problem is underdetermined. So similarly, it shouldn't work for the non linear case in an architecture like the autoencoder, where the problem of finding this reverse mapping is also underdetermined. So what makes it possible for this underdetermined system to learn? $\endgroup$
    – tushar
    Commented Feb 24, 2021 at 9:35
  • $\begingroup$ @tushar it's not a non-linear PCA (doesn't have any PCA features with the exception of dimensionality reduction). Non linear PCA exists, and is not based on neural networks. The decoder is not the inverse of the encoder, that part is right. And the problem is not underdetermined either, because it doesn't want to transpose the encoder weights, rather reconstruct inputs from compressed codes. $\endgroup$
    – Firebug
    Commented Feb 24, 2021 at 12:05
  • $\begingroup$ "reconstruct inputs from compressed codes." why is this not underdetermined then? It's trying to find a mapping from a smaller space to a larger space $\endgroup$
    – tushar
    Commented Feb 24, 2021 at 22:28
  • $\begingroup$ @tushar it might incur a loss, whose severity depends on the compression, yes $\endgroup$
    – Firebug
    Commented Feb 24, 2021 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.