I am studying AutoEncoder to learn how to build a-one-class classification model which is unsupervised learning and I am wondering how to build a-one-class classification model using AutoEncoder. After having done my researches and I got some questions in mind:

  • Does AutoEncoder need backpropagations?
  • Does AutoEncoder require the same weights as in the encoder and decoder parts?
  • If I want to detect outlier ones, do I only train the model with non-outlier ones?
  • After having trained an AutoEncoder, how can I test it as a-one-class classification model?

AutoEncoder is not a classifier, but you can use it as a layer before your classification layers. The reason to use AutoEncoder is to get a better representation of your input, you can think of it as a dimensionality reduction technique like PCA (but a nonlinear one). In order to build an end to end classifier with AutoEncoder you can do the following:

Build the following neural network: Given the input is X, pass it via several layers and output a dense vector D. D will be your dense input representation. Given D you now have two tasks:

  • Classification task: Take D as an input and pass it via several layers with sigmoid as final activation to get the classification output Y*. The loss for this task will be a simple cross entropy loss with the label and the output Y*.
  • AutoEncoder auxiliary task: Take D as an input and try to reconstruct the input X. In order to do it, pass D via several layers and output a dense vector X*, with the same size as X. The loss for this task will be the average square distance between X and X*.

The final loss for your network will be a weighted average between the losses of these two tasks. Backpropagation is used for both tasks. The model validation doesn't change. You still can calculate AUC for your test set using the label Y and the output Y*.

One way to implement it will be the following:

  • Take X as an input, pass it via several fully connected layers and output a latent vector D
  • Take D, pass it via additional fully connected layers and output a dense vector K, with the size of |X| + 1.
  • The last element of the vector K will be your logits. pass it via sigmoid to get the probability Y*.
  • The first |X| elements of vector K will be the reconstruction of X.
  • Create a loss function L which is a weighted average of two losses. The first loss is a simple cross entropy loss between the label Y and the prediction Y*. The second loss is the euclidean distance between X and X*
  • Finally minimize L via gradient descent optimizer
  • $\begingroup$ Do you mean the latent layer by the dense vector D in AutoEncoder? $\endgroup$ – StoryMay Mar 27 at 11:07
  • $\begingroup$ yes. This latent layer is the encoding of your input. $\endgroup$ – ofer-a Mar 27 at 12:09
  • $\begingroup$ As said, I can get the latent features from the latent layer after some iterations of AutoEncoder and use the latent vector as the input features for the classification model but how do I train the classifier for the dataset with only one class? Do I keep the encoding layer and replace the decoding layer with the classification layer with sigmoid function in the output layer and use cross-entropy for the cost function? $\endgroup$ – StoryMay Mar 27 at 12:45
  • $\begingroup$ You keep the encoding layer, and then from this encoding layer you produce 2 outputs. One is your probability prediction (after the sigmoid) and the second is a dense layer X* which is the reconstruction of the input X. Then you combine the two losses, the simple cross-entropy loss and the second loss which is the difference between X and X*. $\endgroup$ – ofer-a Mar 27 at 13:17
  • $\begingroup$ Not to complicate things (ofer-a is exactly right) but you don’t have to produce 2 outputs (in what is called multitask learning, or parallel transfer learning). Instead, you could do one then the other. First train the autoencoder. Then remove the decoder and put the classifier on top of the encoder. Train the classifier. This is called sequential transfer learning. $\endgroup$ – Arya McCarthy Mar 27 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.