13
$\begingroup$

I often deal with reasonable sized amount of time series data, 50-200 million doubles with associated time stamps and would like to visualize them dynamically.

Is there existing software to do this effectively? How about libraries and data formats? Zoom-cache is one example of library focusing on large time series. In Zoom-cache the data is summarized at several resolutions to make it easier to view at different resolutions.

Edit: Also if there is anywhere else I should ask this question or seek an answer please let me know.

$\endgroup$
4
  • 1
    $\begingroup$ Although I have no idea about how it will scale with such huge number of data points, you might want to take a look at Cubism.js. $\endgroup$
    – chl
    Jan 17, 2013 at 9:27
  • $\begingroup$ I took a look at Cubism, which looks good for incremental updating graphs from serial streams of data, but with my short search I did not see anything about caching multiple resolutions or summary data to speed up large data sets. Thats for the pointer though it looks like a cool library. $\endgroup$
    – Davorak
    Jan 17, 2013 at 23:39
  • 1
    $\begingroup$ You might be interested in Zoomdata zoomdata.com/product/fast-data-sharpening-visualization Their pricing is quite high though. $\endgroup$
    – chhantyal
    Feb 20, 2017 at 22:33
  • $\begingroup$ @chhantyal - Thanks! That does look like the kind of tool I was looking for. Though I am no longer working with the same data sets I am still interested in the space so I will probably try out zoomdata's trail at some point. $\endgroup$
    – Davorak
    Feb 22, 2017 at 4:15

3 Answers 3

3
$\begingroup$

Sorry for self-ad, but Thunderplot (thunderplot.com) would be good candidate for it. I wrote it exactly for interactive plotting of large datasets. I didn't test it with 200KK rows datasets, but it works fine with ~10KK rows. Also there's "read every Nth row" feature, so you can reduce amount of data to store/visualize. I can send you the registration key in exchange to one of those large datasets. :-)

thunderplot screenshot

$\endgroup$
1
  • $\begingroup$ thunderplot.com says THUNDERPLOT PROJECT CLOSED THANK YOU FOR YOUR SUPPORT BYE $\endgroup$
    – endolith
    May 4, 2023 at 17:07
2
$\begingroup$

There are a set of research tools called TimeSearcher 1--3 that provide some examples of how to deal with large time-series datasets. Below are some example images from TimeSearcher 2 and 3.

TimeSearcher 2

TimeSearcher 3

$\endgroup$
1
  • $\begingroup$ "November 21, 2005" :/ and if I try to open something, it just locks up. $\endgroup$
    – endolith
    May 4, 2023 at 19:51
1
$\begingroup$

Another self-promoting post because I work for this company, but SensorCloud (sensorcloud.com) uses smart algorithms to graph massive datasets very quickly. It was originally designed with our physical sensors in mind, but it has a CSV uploader to handle any timeseries data.

For example, we uploaded a dataset that had 100 billion data points (over 1 terabyte of timestamp + data values) and you can graph it and interact with it very quickly.

Here's a public link to that dataset: https://sensorcloud.microstrain.com/SensorCloud/data/Z3MFURATHIB8A032/

The link above uses our flash viewer. If you don't want to use flash, here's the javascript viewer: https://sensorcloud.microstrain.com/SensorCloud/data/Z3MFURATHIB8A032/js/

Use the scroll wheel, or Shift+left click to zoom in.

$\endgroup$
1
  • $\begingroup$ So you mean FastGraph? "SensorCloud's Timeseries Data Widget was built from the ground up with speed and data in mind. Our unique algorithms allow you quickly view your data, no matter how large. No other visualization software comes close." $\endgroup$
    – endolith
    May 4, 2023 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.