1
$\begingroup$

I am trying to do something quite unusual: learning a latent representation of some data just by optimizing a decoder. Basically, a probabilistic model of a neural network autoencoder without the encoder.

My approach is to set a prior on the latent codes and then minimize the negative log likelihood via SGD. Ideally this will find the best weights and the best settings for the latent variables.

The purpose is to learn a latent manifold in a simpler way than with an autoencoder.

I think this would work because the lower-dimensionality latent representation, combined with a network with a low number of hidden nodes, will enforce "locality" (is this the correct term?), which means that similar datapoints will be located in nearby values in the lower-dimensional manifold.

In my experiments the latent representations are updated mostly in the very early epochs (first or second), while the weights of the neural network are being updated with a larger degree even in later epochs.

The question is: do you have any suggestions on why this would or would not work? Any pointers in literature that describe similar models and similar phenomena?

$\endgroup$
  • $\begingroup$ I don't understand where the latent representation would come from. It sounds like the output of the network is a prediction/reconstruction of the data, but what are the inputs? $\endgroup$ – user20160 Jun 27 '16 at 15:46
  • $\begingroup$ the inputs are the latent representation. They are set randomly at the beginning and then they are updated along with the weights of the neural network. It's a probabilistic model so it's possible to measure how well the model is fit at any time. $\endgroup$ – fstab Jun 27 '16 at 16:02
  • $\begingroup$ I see. So you'd learn a separate latent representation for each data point, and a common set of decoder parameters? $\endgroup$ – user20160 Jun 27 '16 at 16:07
  • $\begingroup$ exactly. And is the limitedness of the neural network (or at least of the first layers) that actually would allow a "locality" latent representation, which is analogous to techniques used to prevent overfitting (which means, generalizing well, hence having a "simpler" and better positioning in the manifold) $\endgroup$ – fstab Jun 27 '16 at 16:19
  • 1
    $\begingroup$ Learning any kind of latent representation is encoding. $\endgroup$ – Frans Rodenburg Jul 12 at 3:46
1
$\begingroup$

It's an interesting idea. Some miscellaneous thoughts:

If you allow arbitrary mappings from data space to representation, then it seems that your method is more flexible than autoencoders, and includes autoencoders as a special case (i.e. when the mapping exactly matches the encoder portion of an autoencoder). This means that, when an autoencoder can solve the problem, there exists a configuration of your method that can also solve the problem (but doesn't guarantee that this configuration is learnable). It might also be the case that some configurations of your method can solve problems that autoencoders cannot, with the same caveat.

If you have $n$ data points and your latent space has dimensionality $d$, then you'd have $nd$ parameters for the latent representations, plus the parameters of the decoder network. If your original data has dimensionality $d'$, an autoencoder would have $d d'$ parameters for the encoder, plus parameters for the decoder. In the case where you have many more data points than dimensions, your approach may be much more flexible than an autoencoder. This could be beneficial, but also means you might have to worry more about overfitting and possibly runtime. Unlike an autoencoder, it's not obvious that you could escape overfitting by adding more data, since the number of parameters scales with number of data points. It seems like constraints from the 'bottleneck'/decoder would have to be the thing that saves it.

Your optimization problem will most likely be nonconvex, and SGD will converge to a local minimum. My gut feeling is that the results might be highly dependent on the initial configuration, particularly of the initial latent representations.

Compared to autoencoders, your method won't give a mapping from the data space to the latent space, so it can't directly perform out-of-sample generalization (although this may not be needed, depending on your goal). This is common to many nonlinear dimensionality reduction algorithms. If necessary, you could use some auxilliary method to learn such a mapping. It does mean you'd have to think about how to perform validation (e.g. how would you test your algorithm on a held-out data set?).

There's certainly precedent for treating the latent representations of every data point as parameters, and directly optimizing them. Nonclassical multidimensional scaling (MDS) is one example. The objective function is nonconvex, and points are iteratively repositioned in the latent space to find a local minimum. It often helps to initialize with classical MDS (which is convex and can be solved in one shot using an eigendecomposition). Many nonlinear dimensionality reduction methods also treat the latent representations of every point as parameters.

$\endgroup$
0
$\begingroup$

Very interesting idea. There could be some relation to "Gaussian Process Latent Variable Model" (GPLVM) which tries to map from a low dimensional space to the observed space via gaussian processes (instead of neural networks). The reconstruction error is used as objective function while the latent representations are optimized. As far as I know, local minima is a problem but it works in general well for high dimensional data with not so many samples.

Also in some way related is "NeuroScale" which is similar, since it maps from latent space to observed space using a neural network and since the latent representations are found with optimization. The objective is in this case based on the similarity of pairwise distance in latent and observed space.

One major problem with your approach is, that the model is very flexible. Imagine, that you random initialized all latent positions. It is possible that a neural network learns a perfect model without even changing these latent positions. This problem is also known for autoencoders. Here, it is helpful to have an additional regularization condition. Slightly changes in the input should still lead to similar latent representations. This concept can be found in "contractive autoencoders" or in some way in "denoising autoencoders".

I would suggest to train the neural network using dropout and an l2-regularization on the latent space positions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.