Auto Encoders(AE) learn a compressed representation of raw data by trying to reconstruct the input from the hidden representation. On the other hand, Self Supervised Learning(SSL) algorithms learn on the set of auxiliary tasks which expose the inner structure of the data. But one can argue recreation of input also an auxiliary task. So how we differentiate SSL with AE techniques?
$\begingroup$
$\endgroup$
3
asked Nov 2, 2019 at 3:49
-
$\begingroup$ Are you sure that an auxiliary task is required for something to count as self-supervised learning? Do you have a resource backing this up? I am aware that word2vec and autoencoder include solving an aux. task, but there seem to be examples for which it seem that this is not the case: ai.stackexchange.com/a/10624/38174 $\endgroup$– Make42Commented Jun 26, 2020 at 9:35
-
$\begingroup$ yes, I think the main difference of SSL with unsupervised learning is the introduction of auxiliary tasks, which provide a supervisory signal without needing any human input. Also, um agree with your argument on word2vec. $\endgroup$– Shamane SiriwardhanaCommented Jun 28, 2020 at 2:54
-
$\begingroup$ Well, you could also automatically extract a label for supervised learning, from input data without training on an auxiliary task. E.g. you could use the word2vec setup, but not for representation learning. Instead you really want to be able to predict the next word - e.g. in order to be able to build a system that automatically writes text. Now there is no auxiliary task. Would you say this is not SSL anymore? Would you disagree then with ai.stackexchange.com/a/10624/38174 and medium.com/@behnamsabeti/… ? $\endgroup$– Make42Commented Jun 29, 2020 at 13:52
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Yes, both approaches can be seen as doing the same as they are used to learn a representation of an input. But they differ in how the learning is performed. You can consider representation learning part of self-supervised learning (SSL) as an encoding step. In addition to encoding, autoencoders have a decoder too.
VAEs, the most popular encoder assumes representations are distributed according to a prior (e.g., Gaussian) and does (approximate) likelihood maximization. The loss you're trying to minimize is different from usual supervised loss, which is used in SSL, but with self-supervised signals (e.g., rotation etc.).
$\begingroup$
$\endgroup$
Self-supervised learning refers to a really broad collection of models and algorithms. An autoencoder is a component which you could use in many different types of models -- some self-supervised, some unsupervised, and some supervised. Likewise, you can have self-supervised learning algorithms which use autoencoders, and ones which don't use autoencoders.
