How to benchmark performance of an autoencoder An Autoencoder is defined as a device that can extract useful features from data, and also use those features to reconstruct initial data. I'm trying to understand what the word "useful" means in a quantitative manner. Most sources I can find (e.g. Hinton paper) attempt to answer usefulness in a qualitative way. They cluster hidden layer values, color them by some supervised label, and state that the labels look separable, whatever that means.
Let's say for simplicity that I want to train a linear single hidden layer autoencoder on ImageNet or MNIST. I can set the number of neurons in my hidden layer to anything between 1 to the number of pixels in the original image, and even beyond. I would expect the reconstruction error to monotonically decrease with hidden layer size. But I don't know explicitly how much of the data is useful features and how much is not. Can I still benefit in any way from knowing the value of the reconstruction error?
I could further proceed to train a classifier from a hidden layer to the data label, and evaluate the usefulness of the hidden layer by the performance of that classifier. However, this metric is not necessarily specific to the quality of the representation, as it also depends on for example, (a) the intrinsic performance of the classifier (b) potential sensitivity of the classifier to the number of input parameters.
Is there a canonical way to formalize usefulness quantification? My ideas would be to either bypass classification network completely and use something like clustering coefficient within vs across labeled hidden datapoints, or to use some very strictly defined classifier that is somehow guaranteed to be stable.
 A: Testing the strength of the relationship between features and labels sounds a lot like feature selection. Feature selection can be done without building a classifier (e,g, chi-squared tests to remove features which are independent of the label, or remove features with correlation coefficient magnitudes that are too small). These are called "filter-based" methods.
You can also do feature selection by building a classifier (e.g. lasso, boruta) to screen out features which don't improve the model. These are called "wrapper-based" methods.
Unfortunately, comparing two or more auto-encoders won't be as simple as comparing the number of features that your selection method labels "useful." It's conceivable that one autoencoder gives 10 weak predictors while another one gives 1 very strong predictor. You wouldn't necessarily know which predictors and weak and which are strong without assessing their classification performance, and the Question has expressly forbidden that.
There isn't a "canonical" way to do this, because feature selection is a hard, complex task. Beyond "wrapper" and "filter," feature selection methods can be contrasted in terms of computing resources consumed, runtime, suitability of their assumptions, and susceptibility to exclude relevant features or include irrelevant features. It's not possible to summarize all feature selection methods in an Answer; it would be challenging to do so in an academic article.
The Question is profitably reframed in terms of feature selection because there are any number of feature selection methods available to choose from. The only limitation the Question places on methods is that they do not include classifiers, so wrapper-based methods like lasso and boruta are excluded. This is fine. There are lots more.
A: You may view building an Autoencoder is doing a representation learning. For NLP, people are doing sentence embedding learning benchmark.
SentEval: evaluation toolkit for sentence embeddings

We assess their generalization power by using them as features on a broad and diverse set of "transfer" tasks. SentEval currently includes 17 downstream tasks. We also include a suite of 10 probing tasks which evaluate what linguistic properties are encoded in sentence embeddings. Our goal is to ease the study and the development of general-purpose fixed-size sentence representations.

I think this also applicable for your case.

For selecting the layer size, It can be depending on your data size. Say if you have 10K images, may be the embedding size is 64, and if you have 1 million images, the size can be 256.
