Firstly, a (feed forward) neural network can be thought as a lookup table. $f(x) = y$. So it is certainly storing some data.
The theoretical limit of data compression is -$\sum$ p $ln(p)$. The data in a neural network however is stored as weights and biases and not variable length coding, so does this equation even apply?
Say, I have a fully compressed data set of a certain size, will the size of weights and biases be the same if I have perfectly trained the network?
How do I know if my neural net has enough representative capacity or not to represent the entire data set?