4
$\begingroup$

I have a data set that is not too big but high dimensional, let say 10000 dimensional. I want to use an autoencoder to extract relevant features (clusters) in the data. Usually when I have seen autoencoders being employed, the datasets have been rather big, MNIST has about 60000 training and 10000 test observations.

Let say I want to reduce the dimensionality from 10000 to 100 and had only 100 training samples. Wouldn't this lead to that each of the hidden nodes could represent each of my training samples. And how much bigger would my training data need to be to avoid this kind of trivial representation, 1000, 5000 or 10000? Is there some standard test to see if the features extracted are really relevant?

Thanks.

$\endgroup$
2
  • $\begingroup$ Seems like regularized autoencoders is the way to go to prevent learning the identity function also in this case. This chapter from a not yet published book about deep learning has a nice discussion about the topic if someone is interested. deeplearningbook.org/contents/autoencoders.html $\endgroup$
    – johnblund
    Feb 8 '16 at 13:19
  • $\begingroup$ Regularization and/or a small bottleneck dimension (e.g. 10-100). With such a small sample size you may want to do cross-validation, to ensure that features are indeed relevant, and not just because your validation set was the 20 easy samples. Also, plot the reconstructions, as it might reveal a lot about what's going on. $\endgroup$ Nov 26 '19 at 15:55
-1
$\begingroup$

That sounds like an interesting application of autoencoders. In general you want to have at least as many model parameters as you have training examples. It is possible to train a small network (you could use two fully connected layers for instance) in order to accomplish your task, however don't expect it to do well on unseen data from the "same" data distribution.

$\endgroup$
2
  • $\begingroup$ I don't see how this addresses the explicit question. Can you clarify that? Also, did you mean, 'you want to have at least as many training examples as you have model parameters'? $\endgroup$ May 20 '17 at 11:40
  • $\begingroup$ I meant to say that if your model has a large capacity, and you have a training set whose magnitude is much smaller than the number of parameters your model contains...then you will end up memorizing the inputs rather than learning anything substantial, which defeats the purpose of using a neural network to do your dimensionality reduction. Just use PCA. $\endgroup$ Jun 14 '17 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.