I'm trying to find out how many open source projects related to hydrological/environmental modelling are out there, and derive some basic information (e.g. total number, deployments per year, etc.).

Where can I get this info from?

(I have checked OpenHub, GitHub and SourceForge but it seems I cannot download the list of project and related information.)


closed as off-topic by Andy, Xi'an, gung, Nick Cox, kjetil b halvorsen Jan 19 '15 at 15:02

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions about obtaining specific datasets are off-topic on Cross Validated. You are welcome to post this question to the Open Data site instead, thank you!" – Andy, Xi'an, gung, Nick Cox, kjetil b halvorsen
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ I disagree with the decision of putting this post off-topic. Quoting the Help Center > Asking:What topics can I ask about here? collecting data is one of the options. $\endgroup$ – Claudia Jan 19 '15 at 15:09
  • $\begingroup$ Lilith, let me clarify. While "collecting data" is indeed a valid topic on CrossValidated site, it makes more sense to have here data collection questions, focused on statistical perspective of data collection. Open Data StackExchange site, on the other hand, is much better suited for questions on open data sets, so I agree with moderators' suggestion. Unless you have a specific reason to avoid that site, if you'd register on Open Data SE (takes just a moment), this question could be migrated there by moderators and the off-topic flag would be cleared, if I understand correctly the procedure. $\endgroup$ – Aleksandr Blekh Jan 19 '15 at 15:27
  • $\begingroup$ I'm now registered to OD SE and I don't mind if the moderator migrates my question there. $\endgroup$ – Claudia Jan 19 '15 at 15:45

First, a warning from my experience: you will not be able to get meaningful data on open source projects' deployments on a large (geographical) scale. What you can do in that regard, though, is to use number of downloads a very approximate proxy measure for assessing the projects' adoption.

Having said that, I think that you can download some information of your interest from those three project repositories and meta-repositories that you have mentioned, but it's not trivial. You have to use a particular set of APIs for each of them. GitHub data is most likely the easiest to access and their set of APIs is relatively simple. OpenHub (formerly Ohloh) left me with an extremely negative impression in terms of their feedback to researchers that wanted to use their "open" data (haven't used their data set). I've been able to use SourceForge data set for my dissertation research, but even that is available to researchers or students under an academic license. For that, you need to request credentials from a specific person at Notre Dame University, which manages the research-focused version of SourceForge data set, called SourceForge Research Data Archive (SRDA).

As a subset of software for my dissertation's data analysis, I have written some open source R software that works with SRDA (and FLOSSmole meta-repository as well). However, currently the code is not in a form of a standard R package, for various reasons. You are welcome to take a look at it on the following GitHub page: https://github.com/abnova/diss-floss-official (it is not fully and properly documented yet as well). I plan to convert that software into two separate packages - for multi-source open source data collection and for open source ecosystem analysis - if / when I will have a chance. I also recommend you to include CRAN and R-Forge into your data sources, as R ecosystem contains an impressive number of open source software packages in multitude of scientific domains and I would be very surprised, if hydrology is an exclusion from that tradition.

  • 1
    $\begingroup$ Many thanks for your answer Aleksandr! I'm trying to generate a comprehensive review of projects and, at least for the time being I won't focus on one particular programming language. However, that is definitely something to take into account in the future. $\endgroup$ – Claudia Jan 19 '15 at 15:07
  • $\begingroup$ @Lilith: You're very welcome! Feel free to upvote/accept my answer, if it is helpful / you're satisfied. I very much agree with your language-agnostic approach, as I like comprehensive reviews as well. One more thought - it might be easier and more time-efficient to just manually collect data (not scrape!) than to learn several sets of APIs, as I don't expect the number of projects in hydrology to be too high. $\endgroup$ – Aleksandr Blekh Jan 19 '15 at 15:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.