# Lossless Compression Of Data Tables Leveraging Exploratory Analysis?

Lossless compression of data tables seems a natural application of automated modeling based on exploratory data analysis methods. While such automatically generated models are not reliable for statistical inference, they are able to predict the contents of the data table better than general file compression techniques.

The quick and dirty model's information content plus that of residual errors requires far fewer bits than the data tables in their, say, gzipped form.

Are there such utilities available for statistical databases?

• I strongly suggest that you read Hadley Wickham's Tidy Data manuscript. One could also argue that you should use a relational database. I'm sure there are suitable frameworks for such databases that enable a good compression. – Roland Apr 27 '18 at 6:58

• By "zeros" I presume you mean missing data which, in "R" is represented as "NA" (or "NULL"?). Right? So you're saying each matrix would use the GeoCode (which is a positive integer) to index one matrix dimension and the other matrix dimension's index would be the year. Then the stack function composes these 2D matrices into a single "stack" object, indexed by the name of the measure. This object can then be written out to a single .Rdata file with a call to save. Is that correct? – James Bowery Apr 26 '18 at 20:00