2
$\begingroup$

I want to graph some data, containing of an event ID and a date (minimum required resolution is one day). every event ID has multiple dates connected to it, so the min and max date give me a time span. Another useful information would be where in time most of these events took place (like a density graph over this time span).

The problem is, that i have about 1500 different events, with each more than 100 dates connected. In sum i have 1.1 million event/date pairs. (the whole data set is much larger: 200k events with an average of 10 dates)

Are there any good ways to visualize this data? I tried using dot-plots, but it is more a mess and you see more a black wall than any thing else.

If there is no such solution i can break down the data into smaller chunks, but for example the top 10 events (having the most dates connected) sum up to 500k tuples.

If you have concrete examples in ggplot2 or D3.js it would be perfect!

This is an image, how a scatter plot of start date vs end date for the top 1500 events look like. The problem is that most of the events have the same (or similar) span. Sample Scatter plot of the top 1500 events

This is the same plot, but with more data. As you can see, the less common events have shorter span. Sample Scatter plot of the top 10000 events

Also the same data as in the last plot, but with a marker for the count of dates inside the span. As you can see there are two big blobs and the rest is quite small. scatter plot with count of events in span as size

now a scatter plot for all the data, you can clearly see there is a line with lots of events with only one date enter image description here

and as last image, this is the whole data set as density2d: density2d map of the whole data set

and the same for all events with more than 10 dates: enter image description here

The problem with the last two plots is, that they do not count the occurences inside the span. they just show me where a lot of time spans are. How can i put the count of events inside the time span also into this calculation?

$\endgroup$
  • $\begingroup$ You can set the fill attribute for the contour plots, which looks nicer. You can also try to apply logarithmic scaling to the density to compensate the extreme densities for the singly day events. Have a look at this: stackoverflow.com/questions/21273525/… $\endgroup$ – ziggystar Jul 7 '14 at 8:05
  • $\begingroup$ @ziggystar the links looks promising, but if i apply all values to the density (so that a real density is generated) my computer hungs up for some minutes before i can see the whole plot because there are just too many values (>1.5 million) - i mean it looks very promising but is too slow $\endgroup$ – reox Jul 7 '14 at 11:02
  • $\begingroup$ Then try subsampling the data. $\endgroup$ – ziggystar Jul 7 '14 at 12:18
  • $\begingroup$ Another (general) scatterplot option is to graph the length of the time interval on the Y axis and the midpoint of the dates on the X axis. This allows you to draw a triangle to the begin and end dates. What is the end goal of the analysis? In terms of R hanging it is often better to do the number crunching outside of ggplot2 and then plot the summaries (often other plotting devices are much faster). $\endgroup$ – Andy W Jul 7 '14 at 12:52
  • $\begingroup$ I've used scatterplots similar to yours in the past, and I would recommend putting the min/start date along the y-axis (I've found it's even better when in descending order), and the end date along the x-axis. This makes for more intuitively understandable visual lines, where events starting on the same day are a visual "row" with a width that corresponds to their duration (useful since people naturally read graphs with time along the x axis) $\endgroup$ – Fabio Beltramini Jul 7 '14 at 14:40
4
$\begingroup$

It's not clear why you're graphing min versus max. Just to reduce complexity? It can be expected that looking at min vs. max for the top N will produce a plot with low min values and high max values. And min vs. max for all events will follow y=x since most events have a single occurrence.

What is the event/date relationship. Is it really one event that spans multiple (possibly consecutive dates)? Or is it a group of otherwise-unrelated events that have the same descriptor? The former may justify min/max visualizations.

In any case, you may find a set of histograms useful, especially with the 1-D striping seen in the scatter plots. Histogram of all dates, just the mins, just the maxes, just the lengths (max-min), just the dates of the top N events (one histogram per event, that is).

If you find a meaningful pair of variables (and maybe it's min and max), then a 2D heatmap may be a tidier equivalent to a scatter plot.

enter image description here

$\endgroup$
1
$\begingroup$

Since every event corresponds to two variables (begin and end), I'd also try a scatter plot.

One possibility is to plot begin date on one axis and end date on other axis. A different plot would show begin on one axis and span on the other.

If you have too many points you can either

  • decrease ink amount with same data: smaller point size, or more transparency (alpha in ggplot)
  • decrease ink amount by subsampling (display only a random subset of the data)
  • or abstract the data: use 2d density estimates (density2d in ggplot).

If you are not interested in the interaction between start and end date, you can also visualize the distribution over e.g. start date through use of a histogram or density plot.

$\endgroup$
  • $\begingroup$ yes, i thought about that too... i probably need two graphs, one show the span and one show the distribution. The optimal way would be to have both in one chart, so i can see which events have a wide span and homogenious distribution and so on $\endgroup$ – reox Jul 7 '14 at 7:21
  • $\begingroup$ @reox This sounds like you want to cluster your data somehow (splitting it into groups). This is more complicated and probably requires additional variables to work well. If you provide a sample plot of your data, you might get better feedback. $\endgroup$ – ziggystar Jul 7 '14 at 7:27
  • $\begingroup$ i added some graphs i just made from the data. $\endgroup$ – reox Jul 7 '14 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.