53
votes
$\begingroup$

EDIT: The Web Technologies and Services CRAN task view contains a much more comprehensive list of data sources and APIs available in R. You can submit a pull request on github if you wish to add a package to the task view.

I'm making a list of the various data feeds that are already hooked into R or that are easy to setup. Here's my initial list of packages, and I was wondering what else I'm missing.

I'm trying to limit this list to "real time" or "close to real time" data feeds/APIs, where the underlying data might change between downloads. There's plenty of lists out there for static datasets, which only require one download.

This list is currently biased towards financial/time series data, and I could use some help expanding it into other domains.

Free Data:
Data Source - Package
Google Finance historical data - quantmod
Google Finance balance sheets - quantmod
Yahoo Finance historical data - quantmod
Yahoo Finance historical data - tseries
Yahoo Finance current options chain - quantmod
Yahoo Finance historical analyst estimates - fImport
Yahoo Finance current key stats - fImport - seems to be broken
OANDA historic exchange rates/metal prices - quantmod
FRED historic macroeconomic indicators - quantmod
World Bank historic macroeconomic indicators - WDI
Google Trends historic search volume data - RGoogleTrends
Google Docs - RGoogleDocs
Google Storage - RGoogleStorage
Twitter - twitteR
Zillow - Zillow
New York Times - RNYTimes
US Census 2000 - UScensus2000
infochimps - infochimps
datamarket - rdatamarket - requires free account
Factual.com - factualR
Geocode addresses - RDSTK
Map coordinates to political boundaries - RDSTK
Weather Underground - Roll your own
Google News - Roll your own
Earth Sciences netCDF Data - Roll your own
Climate Data - Roll your own
Public health data - Roll your own
OAI Harvester - Open Archives Initiative harvester
RAmazonS3 - S3 Amazon storage server
Rflikr - Flikr api

Requires a subscription:
Bloomberg - RBloomberg
LIM - LIM
Trades and Quotes from NYSE - RTAQ
Interactive Brokers - IBrokers

$\endgroup$
7
  • 1
    $\begingroup$ depends on what you mean by 'easy to set up'... I have a number of custom scripts that are 'easy to set up' $\endgroup$
    – David LeBauer
    Commented Jul 5, 2011 at 19:44
  • 2
    $\begingroup$ Right now I'm defining "easy to setup" as "the package exists" or "someone's posted/published a function to download the data" $\endgroup$
    – Zach
    Commented Jul 5, 2011 at 20:07
  • $\begingroup$ I would further define "the packages exists" as "the package is on CRAN" or "the package is on R Forge," "the package is on some CRAN-like, public repository." $\endgroup$
    – Zach
    Commented Nov 10, 2011 at 17:05
  • $\begingroup$ Addition: Federal Register API (in the works). $\endgroup$
    – Fr.
    Commented Dec 2, 2012 at 10:54
  • $\begingroup$ This thread may also be of interest: Import stock price from Yahoo Finance into R. $\endgroup$
    – gung - Reinstate Monica
    Commented Dec 19, 2012 at 2:42

3 Answers 3

Reset to default
8
votes
$\begingroup$

  1. Instructions for using R to download netCDF data can be found here, a common format used for storing Earth science data, e.g. as in marine geospatial data from OpenEarth or climate model driver and forecasts from UCAR

  2. rnpn (under development) enables you to get data from the National Phenology Network - a citizen science project to track the timing of plant green-up, flowering, and senescence. See the developer's blog post.

  3. -obsolete- RClimate provides tools to download and manipulate flat-file climate data (with tutorials, including here-

  4. Download historical finance data with tseries::get.hist.quote

  5. Michael Samuel's documents downloading public health data

  6. raster::getData provides access to climate variables via worldclim

$\endgroup$
5
  • $\begingroup$ What is "netCDF data?" $\endgroup$
    – Zach
    Commented Jul 6, 2011 at 13:16
  • $\begingroup$ @Zach NetCDF is a kind of universal structured way of storing large datasets. An alternative approach relies on the HDF5 format. $\endgroup$
    – chl
    Commented Jul 6, 2011 at 19:58
  • $\begingroup$ @David (+1) Many thanks for the fourth point! $\endgroup$
    – chl
    Commented Jul 6, 2011 at 19:59
  • $\begingroup$ What happend to RClimate? I cannot find the package anywhere $\endgroup$
    – Daniel Ryback
    Commented Mar 16, 2014 at 20:20
  • $\begingroup$ now there is a ncdf4 library on R to handle netcdf4. It is more complicated to install than ncdf but the later is deprecated I think. A few years ago I even managed to install netcdf4 with Min GW64 to make this package work on windows :) if I did it, it means it's feasible ... and it seems that D. Pierce (author of the package) is providing sources on his web site cirrus.ucsd.edu/~pierce/ncdf $\endgroup$
    – robin girard
    Commented Oct 28, 2014 at 21:52
8
votes
$\begingroup$

There's a project aimed at creating R packages with this objective (R interface to real-time APIs) called rOpenSci, which has 18 packages currently available or in development. Some (rnpn, rfishbase) are on your list already.

Great list! and full disclosure - I'm part of the rOpenSci project.

$\endgroup$
1
vote
$\begingroup$
  1. ONETr - efficient interaction with the O*NET™ API, offering occupational descriptor data from the U.S. Department of Labor.
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.