# Any suggestions for storing diverse data sets for on-line visualization?

We are trying to design a data repository for storing datasets, from many different sources, in such a way that they can easily be used in online data presentations.

This data repository will be part of a website that reviews, interprets and discusses scientific reports and data on public health. It is targeted at scientists, medical experts and policy makers.

A prominent aspect of the site will be data visualizations: charts and maps. Data used in visualizations must be presented in tabular format and be available for download, as well. We are considering to use a JavaScript library for the charts, possibly high charts. Maps will probably be mainly images with click area's, but some server site tool may be used for specific maps.

The data that we use, is collected from other organizations, in the form of database dumps and ASCII files, or sometimes simply copied from scientific studies. We are considering to link to open data sources, in the future.

Previously, we have created a generic data model in which each data set was stored in the same generic format: a data set was linked to a variable and for each measure the following dimensions were available: period, area, age, sex and social economic status, all of them optionally present in a data set. Finally, the measures were stored as floating point numbers. As an example, think about mortality of coronary heart diseases. A record in the dataset would be the absolute number of deaths in the year 2014 for Dutch males in the age group 65 to 75. Meta data describing the complete set, certain domain values of a domain or individual records were supported. Domain values, like 2011, male and 65-75 were shared between data sets. This allowed for searching across data sets and, to some extent, facilitated combining different data sets in one visualization. This data model, however, required extensive preparation of data before it could be imported, including un-pivoting and mapping of domain values.

Another aspect was versioning of data sets. In some cases, the most recent available data should be used, in other cases, a fixed set is to be used, even if more recent data is loaded into the repository. In that case, each visualization is linked to a specific version. This version is only upgraded if the text accompanying the visualization is reviewed and updated as well. Imagine a trend chart in which the latest years show some new trend. This newer data is to be shown only when the editor has done an analysis on that trend.

So, this is not so much about large data sets, but rather about large numbers of diverse data sets. Characteristics:

• The presence of various measures. Sometimes, more than one measure was present, for instance an absolute number as well as a percentages or a standardized rate or confidance intervals.
• Various aggreagation levels in the dimensions: years, months, days;
5-year age groups or 0-65/65+, etc.
• Meta data and footnotes on sets, dimensions and/or single values. It must be possible to show these in tabular presentations with footnote symbols linking to descriptions.
• Versioning of data sets, where newer versions remove, correct and/or add values.
• Aggregation may or may not be trivial. For absolute numbers,
aggregation is a summation. This is easily performed and will be
supported. For standardized rates, aggregation is complex and outside the scope of the repository. In such case, pre-aggregated totals must be added to the dataset, to be able to show marginals in cross-table presentations. These totals must be distinguishable from base
numbers, as it must be possible to leave them out of visualizations. Imagine a stacked bar chart over years, where the series are age
groups. A total over all ages must not be included in the stacked
bars.
• We must be able to back-track data used in presentations to the
original source, to be able to explain were they came from. So
ideally, nowhere in the work-flow, data has to be copied and pasted nor edited manually.

On top of the data repository, there will be a presentation composer. This composer must be capable of querying the repository. These queries should ideally be generated from choices the editor makes: filtering, selecting dimensions as x-axis, series, etc. Linking to a newer version of a data set should be supported, without completely redefining the presentation.

At this point, we consider supporting various of the larger datasets with a dedicated data model, and some sort of adaptors for the presentation composer, mainly to avoid various issues with importing data sets in a generic model.

Does anybody recognize this scenario and have some suggestions about best practices or maybe even an out-of-the box solution?

• 787 words!? Is there any way you could condense that to the essential elements in a way that it might still be useful for future readers? – Anko Feb 19 '14 at 16:30
• The question looks more like a specification of project, with many subquestions... – Piotr Migdal Feb 20 '14 at 13:07

As a generic answer, if you main goal is to visualize date with JavaScript, go with JSON (that is, lists and dictionaries, nested as many times as you want) as they are JavaScript object you will need, at one stage or the other.

For small data (say, up to 1MB on client side and up to 100 MB on the server side) you can just put it into a static json file.

For larger (or when you want to add, update or remove items), it depends on the exact structure of your data. If you can easily tabularize it, some variant of SQL will do the job.

If it is not, or is very un-normalized, or you want to store JSON object (rather than generate them on spot), MongoDB may be the option.

Schreurs, as someone tolds it looks more like a specification of project, with many subquestions. But It is very interesting, because I am working in the same direction, it is a datawarehouse for public health too and bioinformatics. Well, in a simple way I have this pipeline:

1-Users enter data in a web form.

2-Data are stored in a database mysql.

3-Users require information. Select data and time for analysis.

4-I use PHP to retrieve all necessary data to the analysis process from MYSQL.

5-PHP sends data to R using shell_exec(...).

6-R do the job it must to do. Generate matrix or vectors.

7-R sends those processed datas to PHP thrugh JSON.

8-PHP retrieve this JSON object and sends do JAVASCRIPT trough jQuery.

9-In javascript environment jQuery use the data to create or generate charts trough d3Js library, and show them in the browser.

10-Finally I use bootstrap3 as template to the chats and maps.

11-There is a lot of possibilities to generate charts in web browser. I use d3js .I can change the code as I need. I use SVG to customize the charts or tables.

12-Sometimes I need to show the information in maps (like some diseases outbreaks). For that I use qGis to create the .shp files I need. Then I can manage the informatin I want to show in maps, using PHP-(JSON) and send them to jvector maps through jquery

13-Bootstrap and jvectormap are ready to be displayed in mobiles devices.