2
$\begingroup$

Lossless compression of data tables seems a natural application of automated modeling based on exploratory data analysis methods. While such automatically generated models are not reliable for statistical inference, they are able to predict the contents of the data table better than general file compression techniques.

The quick and dirty model's information content plus that of residual errors requires far fewer bits than the data tables in their, say, gzipped form.

Are there such utilities available for statistical databases?

$\endgroup$
  • $\begingroup$ I strongly suggest that you read Hadley Wickham's Tidy Data manuscript. One could also argue that you should use a relational database. I'm sure there are suitable frameworks for such databases that enable a good compression. $\endgroup$ – Roland Apr 27 '18 at 6:58
0
$\begingroup$

My experience lies in sparse matrices for optimization. Hopefully somebody more knowledgeable will come along to replace my answer.

I took a quick look at sparse matrices in R and found this: http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/

Potential for 1000-fold storage reduction - depending upon proportion of zeros.

As to approach, I would store the data for each metric in its own sparse matrix. Each matrix then consists of a series of entries: (country, year): observation. Your data file would consist of a stack of sparse matrices, one for each metric you are storing.

$\endgroup$
  • $\begingroup$ By "zeros" I presume you mean missing data which, in "R" is represented as "NA" (or "NULL"?). Right? So you're saying each matrix would use the GeoCode (which is a positive integer) to index one matrix dimension and the other matrix dimension's index would be the year. Then the stack function composes these 2D matrices into a single "stack" object, indexed by the name of the measure. This object can then be written out to a single .Rdata file with a call to save. Is that correct? $\endgroup$ – James Bowery Apr 26 '18 at 20:00
0
$\begingroup$

The answer is "Semantic Compression" of relation tables. The most recent work in this area is the 2018 PhD thesis "EXTRACTING AND UTILIZING HIDDEN STRUCTURES IN LARGE DATASETS".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.