How to extract the features making up the hidden layers in Autoencoders

Question

Apologies if this question has been asked before but I haven't come across any so far.

I have been experimenting with Autoencoders using Keras and Theano as my back end based on the tutorial from this tutorial which has been going on well so far. So I have a high-dimensional (over 15K features) dataset as examples with no labels and I would like to reduce these into some manageable features which I presume are in the hidden layer of an Autoencoder neural network. My question, therefore, is how would one extract the input features that are contributing to each of features in the hidden layers as I would like to use these for further analysis.

Thanks in advance

Answer 1

See this Feature Selection Guided Auto-Encoder.

They proposed a framework to select informative features. In this framework, the discerning hidden units were distinguished from the task-irrelevant units at hidden layer, and the regulariser on the selected features in turn enforces the encoder to focus on compress important patterns into selected units.

Citation

Wang, S., Ding, Z., & Fu, Y. (2017). Feature Selection Guided Auto-Encoder. In AAAI (pp. 2725-2731).

Answer 2

Most neural-network based autoencoders I am familiar with do not make a hard decision on which input features will be kept and which will be thrown away. Think of a typical Fully Connected autoencoder. All of the inputs will be connected to all nodes of the hidden layer representation.

If you wanted to try and define some sort of feature importance in the fully connected case, you could look at the sum of the absolute value of the weights going out of each feature node, but this relies on all of your input features having a comparable scale. If, for example, one feature is height of person in feet and another feature person's income in USD, these will likely have very different weights regardless of importance (and your neural network will likely not train well either).

You could try to encourage disparities in weights through weight regularization. In this case, I'd use L1 before I'd try L2. See When will L1 regularization work better than L2 and vice versa? for more details.

Defining feature importance will be much more difficult in the case of a convolutional neural network (or pretty much any novel, non-Fully Connected based architecture).