I have decided to improve my well-being and in case being successful write a note about this and share it with my peers for free and try to help them improve there well-being as well. But on this road, I need foundational advice from the database practitioners.

Background Description:

In my research, I need to collect a lot of different types of time-series (GDP, Real GDP, Nominal GDP, Consumption, Investment, Percapita GDP, Number of Hours Work, Unit Labor Cost and many more). Right now, I am following and collecting about 155 time-series. Until now, I have been updating them manually as soon as an update was published on statistical web pages. I have been doing this using Excel spreadsheet (like, downloading new excel and then copying and pasting the necessary data point into my excel). However, this is a very daunting task, and it is prone to a lot of errors. Checking and finding the error is another nightmare. So, in total, maintaining this Excel database (updating checking and documenting) takes about 10-12 hours per week approximately.

My Task:

  1. I want to create a database where I will have all these and many more variables (I want each variable to have a name, description, source description (ideally with URL included), date of last update etc.) stored.
  2. I also want to be able to update the database automatically from downloaded Excel or CSV file (These files come in different formats from various webpages. Some are structure vertically some horizontally).
  3. I also want to have some error checkers for easier debugging the base.
  4. The series are of different frequencies (some are daily, some weekly, some monthly and some quarterly and yet some annual). I want to be able to easily convert from daily to monthly or from monthly to quarterly or vice versa using my own predetermined formula. (being it average or median or whatever the current task requires)
  5. I also want to then later to be able to easily query any variable or several of the variables and create a dashboard in various programs like MATLAB or Python or Julia to visualize dynamics and use them for regression or model estimations in these programs.


For all the tasks described above which database management program would you recommend (ideally, it would be better to be it free and open source)

P.S. I tried writing MATLAB script for automatic updates, but it's very inconvenient.

P.P.S The question has been also cross-posted on Database Administrators.

Thanks, Giorgi.

  • $\begingroup$ I think this belongs on another Stack Exchange site, though not sure which one. I guess one on programming or databases. $\endgroup$ Dec 17 '20 at 7:48
  • 1
    $\begingroup$ Thanks. But I though on this site I might find someone who has already thought and tried database through this task. $\endgroup$
    – G.T.
    Dec 17 '20 at 7:54
  • 1
    $\begingroup$ @G.T. Try to cross-post the question also on dba.stackexchange.com by indicating a link on this one there and vice versa. $\endgroup$ Dec 17 '20 at 8:12
  • 2
    $\begingroup$ Have you seen db.nomics.world? Don't use excel sheets, except for presenting data, do all manipulations in Python. $\endgroup$
    – Konstantin
    Dec 17 '20 at 8:31
  • 1
    $\begingroup$ I checked it and it works for developed countries but for a small country Georgia it misses a lot of series. It tries to Fetch them through WEO which is upodated 2 or 3 months after the official statistics is published on Statistical Office of Georgia. So, I prefer to have my onw construction. $\endgroup$
    – G.T.
    Dec 17 '20 at 8:52