I believe this is actually related to the topic of "auto-encoding" but my question is trying to express some basic gaps in my knowledge that I have not seen directly addressed anywhere.
You can train a Neural Network to learn the Identity Function (f(V)=V) on some dataset. If the network is smaller than the input and output vectors, then it behaves as a lossy compression algorithm. Are there any empirical or theoretical studies about what happens as you play with the parameters of this system?
I would expect to find some empirical data on how different real-world datasets require different network sizes, and also some theorems about upper or lower bounds in network parameters and a parametric model of network performance based on statistical properties of the dataset.
Also, if you take such a network optimally trained on the Identity Function, how does it perform compared to something like arithmetic coding for compression, or conversely how does arithmetic coding compare to such an autoencoding scheme for use as the pre-training step of a deep learning stack? (if this last paragraph sounds absurd, google "data compression and machine learning")
My intuitive understanding of arithmetic coding (LZMA), is that it learns to predict the next character in the stream, and then uses some bits to flag the spots where the prediction is wrong and needs to use the 2nd or 3rd or Nth most likely prediction instead. If you remove these extra bits then it should behave just like an RNN in compression (learning mode) and uncompression (generative mode). Is this correct?