An Autoencoder is defined as a device that can extract useful features from data, and also use those features to reconstruct initial data. I'm trying to understand what the word "useful" means in a quantitative manner. Most sources I can find (e.g. Hinton paper) attempt to answer usefulness in a qualitative way. They cluster hidden layer values, color them by some supervised label, and state that the labels look separable, whatever that means.

Let's say for simplicity that I want to train a linear single hidden layer autoencoder on ImageNet or MNIST. I can set the number of neurons in my hidden layer to anything between 1 to the number of pixels in the original image, and even beyond. I would expect the reconstruction error to monotonically decrease with hidden layer size. But I don't know explicitly how much of the data is useful features and how much is not. Can I still benefit in any way from knowing the value of the reconstruction error?

I could further proceed to train a classifier from a hidden layer to the data label, and evaluate the usefulness of the hidden layer by the performance of that classifier. However, this metric is not necessarily specific to the quality of the representation, as it also depends on for example, (a) the intrinsic performance of the classifier (b) potential sensitivity of the classifier to the number of input parameters.

Is there a canonical way to formalize usefulness quantification? My ideas would be to either bypass classification network completely and use something like clustering coefficient within vs across labeled hidden datapoints, or to use some very strictly defined classifier that is somehow guaranteed to be stable.

  • 1
    $\begingroup$ Your question title is how to select layer size, but the contents suggest you are asking how to benchmark the embedding. $\endgroup$
    – Haitao Du
    Jul 8, 2020 at 12:41
  • $\begingroup$ "But I don't know explicitly how much of the data is useful features and how much is not... Is there a canonical way to formalize usefulness quantification?" Useful for what purpose? An AE is only trained to reconstruct the input data. Whether or not that reconstruction is helpful for any particular task depends on the data and the task. This is the same for other representation tasks like PCA. Indeed, your example of a linear AE makes your question closely linked to PCA because a linear AE is a PCA that does not enforce orthogonality of the new basis. $\endgroup$
    – Sycorax
    Jul 8, 2020 at 12:50
  • $\begingroup$ @HaitaoDu I will consider renaming the post, should I try to call it "Benchmarking the AE representation?" $\endgroup$ Jul 8, 2020 at 12:52
  • $\begingroup$ @Sycorax This is part of the problem - the "particular task" is typically about looking and hoping to see some pattern emerge - not a very formal approach. I am interested in answering the question on how to test if a certain autoencoder architecture works, and if it works well. I have also given an example - please tell me how to robustly test how well the latent representation preserves the ability to discriminate labelled data, and how to use that test to determine the size of the latent representation. I am aware on relationship to PCA. Please ignore the shallow linear part. $\endgroup$ Jul 8, 2020 at 13:01

2 Answers 2


Testing the strength of the relationship between features and labels sounds a lot like feature selection. Feature selection can be done without building a classifier (e,g, tests to remove features which are independent of the label, or remove features with correlation coefficient magnitudes that are too small). These are called "filter-based" methods.

You can also do feature selection by building a classifier (e.g. , ) to screen out features which don't improve the model. These are called "wrapper-based" methods.

Unfortunately, comparing two or more auto-encoders won't be as simple as comparing the number of features that your selection method labels "useful." It's conceivable that one autoencoder gives 10 weak predictors while another one gives 1 very strong predictor. You wouldn't necessarily know which predictors and weak and which are strong without assessing their classification performance, and the Question has expressly forbidden that.

There isn't a "canonical" way to do this, because feature selection is a hard, complex task. Beyond "wrapper" and "filter," feature selection methods can be contrasted in terms of computing resources consumed, runtime, suitability of their assumptions, and susceptibility to exclude relevant features or include irrelevant features. It's not possible to summarize all feature selection methods in an Answer; it would be challenging to do so in an academic article.

The Question is profitably reframed in terms of feature selection because there are any number of feature selection methods available to choose from. The only limitation the Question places on methods is that they do not include classifiers, so wrapper-based methods like lasso and boruta are excluded. This is fine. There are lots more.

  • $\begingroup$ I'm sorry, this answer is too brief to be of any use. You said 3 words: chi-squared, lasso, boruta. The first one is a test - what do I apply it to? The latter two are feature selection algorithms based on regularization or some such - I do not see how that helps answer the question. I do not want to construct an algorithm that is better at feature selection. I want to take an existing algorithm of feature selection and evaluate its performance $\endgroup$ Jul 8, 2020 at 13:53
  • $\begingroup$ Feature selection is a large topic. There are lots of different methods with varying degrees of complexity, simplifying assumptions, and performance profiles. I couldn't summarize all of it in an Answer. If you'd like to learn more about feature selection methods and topics, you might conduct a search of stats.SE, or review posts bearing the tag feature-selection. A literature review of feature selection could also be a helpful place to start. $\endgroup$
    – Sycorax
    Jul 8, 2020 at 14:04
  • $\begingroup$ You're correct that all of these methods are used in different ways, with different assumptions. My point is that feature selection, as a topic, is wide-ranging and diverse. I'm aware that classification methods are out-of-scope of your interest. Boruta and Lasso are included as illustrative examples of how broad a topic feature selection is: it ranges from Stats 101 material like a $\chi^2$ test to more advanced methods. $\endgroup$
    – Sycorax
    Jul 8, 2020 at 14:07
  • $\begingroup$ It's ok, thanks for your effort. My problem really is that I come from direction of computational neuroscience. The actual question is whether a model of some circuit in the brain is effectively an autoencoder, and how well does it work. Even if people have ideas on approximately what task the circuit solves, there is no way to know how it does it without making assumptions. The dream is to have a generic metric to quantify to what extent a certain dynamical system acts as an unsupervised learning device. But that is wishful thinking. Maybe I quit neuro and go work for an easier field :D $\endgroup$ Jul 8, 2020 at 14:21
  • $\begingroup$ I don't see any reason to quit neuroscience. Obtaining knowledge from data is hard; everyone would love for there to be one model that's always correct for every problem. But that model provably doesn't exist. So we've got to do what we can with the methods that are available, and try to invent better tools where we can. In other words, we've got to engage in science to obtain knowledge. $\endgroup$
    – Sycorax
    Jul 8, 2020 at 14:34

You may view building an Autoencoder is doing a representation learning. For NLP, people are doing sentence embedding learning benchmark.

SentEval: evaluation toolkit for sentence embeddings

We assess their generalization power by using them as features on a broad and diverse set of "transfer" tasks. SentEval currently includes 17 downstream tasks. We also include a suite of 10 probing tasks which evaluate what linguistic properties are encoded in sentence embeddings. Our goal is to ease the study and the development of general-purpose fixed-size sentence representations.

I think this also applicable for your case.

For selecting the layer size, It can be depending on your data size. Say if you have 10K images, may be the embedding size is 64, and if you have 1 million images, the size can be 256.

  • $\begingroup$ Thanks for your answer. I will have a read on the link and get back to you I am a bit puzzled by your second statement. Is it guaranteed that the optimal embedding size grows with the number of images? It should depend on the size of the explored phase-space. So if I keep adding new objects, it would grow, but if I keep adding images of the same cat from different angles, at some point new images are already well-represented by learned distribution and the hidden layer size should not grow any more. Is this correct? $\endgroup$ Jul 8, 2020 at 13:09
  • $\begingroup$ @AleksejsFomins think about "entropy". if dim size is too small, say 4, how can be used to represent large amount of data? $\endgroup$
    – Haitao Du
    Jul 8, 2020 at 13:18
  • $\begingroup$ If the data is a binary variable between zero and one you only need 1 number to represent it - the probability of seeing a 1. It does not matter how many datapoints you have. Same applies to much larger classes of problems. Say there are only 3 images, but all of them are noisy. Again, your hidden layer size would not scale with the number of noisy images. The same is probably true with 3D rotations of an object - at some point the ability of NN to interpolate is so good that additional datapoints do not add new information $\endgroup$ Jul 8, 2020 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.