Autoencoder for sparse data

Question

Suppose I have a big (1,000x20,000) sparse (95% of elements are zeros) matrix with counts. I want to use autoencoder to encode-decode this matrix. How should I do it? Are there any tricks or guidelines to do it efficiently?

Thank you

Answer 1

I never used autoencoders for sparse data, but my first reaction was "why should this matter?". I found the question interesting, so I made a small Google search and among the first search results, I found an answer by Ian Googfellow to a similar question, who says

You shouldn't need to do anything special for this. Standard good practices for initialization and training should take care of it.

I guess, this could be considered as an authoritative answer for your question.

I don't know how other frameworks, but Keras and TensorFlow support sparse matrices (if it is a matter of memory performance). All you need is a dense, convolutional, or recurrent (depending on nature of your data) layer, or layers, as encoder, and same things for decoder, where on the output layer you would need something like exp function to transform the outputs to non-negative values, if it is counts data (think of Poisson regression).

Answer 2

(Apologize in advance for quite late response)

To my knowledge, for very sparse data you may want to first try out Truncated Single Value Decomposition (SVD), which is implemented in scikit-learn python library. See sklearn documentation / user guide for detail.

There seems to be some research in using Autoencoders for sparse data. This paper proposes a seemingly simple, python-implemented algorithm, and shows it is superior to existing methods including SVD.