1
$\begingroup$

I will be collecting huge amounts of computer performance data and the datasets will get large very quickly.

This necessitates reducing the size of the data by compressing the historical data using aggregation techniques. This allows historical data to be available for comparison, and keeps the data size manageable.

With time series data what kind of aggregation techniques are advisable. Are there any useful resources (textbooks, papers, articles etc.) that discuss this?

Initial thoughts are measures of central tendency like mean, median. Measures of dispersion like stddev etc and frequency histograms.

Any others. What kind of periods should be aggregated over? Hourly, daily etc.? What level of detail allows useful comparison?

Thanks

$\endgroup$
1
  • $\begingroup$ keywords: compressed sensing. $\endgroup$ Feb 14, 2016 at 2:54

1 Answer 1

1
$\begingroup$

If the context of your data affords a means to divide it into similarly-sized data buckets, often by timestamp, then two possible solutions you might consider are: Hadoop (with Spark for streaming data, with Hive, using Java or MapReduce, and optionally Hbase database) or MapR. Hadoop/Spark and MapR offer timestamp partitioning solutions that can quickly be searched and analyzed within seconds to hours instead of hours to days. MapR's website offers an insightful book with large database time series examples, written by Ted Dunning and Ellen Friedman for free. See: "It's About Time: Time Series Databases", at: https://www.mapr.com/time-series-databases-new-ways-store-and-access-data?cid=blog.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.