1
$\begingroup$

In Page 12 of this paper: https://arxiv.org/pdf/1511.05644.pdf, they mentioned that a drawback of non-regularized autoencoders is:

"but it is widely known that non-regularized autoencoders “fracture” the manifold into many different domains which result in very different codes for similar images [Hinton]"

They did not provide reference in the paper and I searched it in Google but did not find related information. Could anyone provide any links or papers on this issue?

Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The issue of manifold fracturing is mentioned in this set of lecture notes by Hinton:

University of Toronto Computer Science 
CSC 2535: 2013 
Lecture 11 
Non-linear dimensionality reduction 
Geoffrey Hinton

See slides 73-79

https://www.cs.toronto.edu/~hinton/csc2535/notes/lec11new.pdf

Synopsis of slides 73-79: Unlike PCA, nonlinear autoencoders can potentially fracture a manifold into many different domains, leading to very different codes in the embedding layer for nearby data points in the original space.

Hinton offers Stochastic Neighbor Embedding as a regularizer that would make it costly to fracture the manifold: "Its like putting springs between the codes to prevent the codes for similar datapoints from being too far apart."

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Your answer should contain more than just a link. $\endgroup$
    – Michael R. Chernick
    Commented Dec 8, 2017 at 5:56
  • $\begingroup$ Ok, thanks. Didn't realize this applies when a person only asks for references rather than an explanation. $\endgroup$
    – Jay Reynolds
    Commented Dec 9, 2017 at 17:10
  • $\begingroup$ One rationale is that links can die but CV wants to preserve the questions and answers. $\endgroup$
    – Michael R. Chernick
    Commented Dec 9, 2017 at 17:49
  • $\begingroup$ Take my suggestion to elaborate because apparently the OP is very happy with your answer. Simply edit the answer to give a complete citation for the reference. $\endgroup$
    – Michael R. Chernick
    Commented Dec 9, 2017 at 17:58
  • $\begingroup$ Michael, may I ask for one more review? I appreciate your guidance on my first attempt at an answer here. $\endgroup$
    – Jay Reynolds
    Commented Dec 9, 2017 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.