103
votes
$\begingroup$

I've been working on a new method for analyzing and parsing datasets to identify and isolate subgroups of a population without foreknowledge of any subgroup's characteristics. While the method works well enough with artificial data samples (i.e. datasets created specifically for the purpose of identifying and segregating subsets of the population), I'd like to try testing it with live data.

What I'm looking for is a freely available (i.e. non-confidential, non-proprietary) data source. Preferably one containing bimodal or multimodal distributions or being obviously comprised of multiple subsets that cannot be easily pulled apart via traditional means. Where would I go to find such information?

$\endgroup$
2

25 Answers 25

Reset to default
43
votes
$\begingroup$

Also see the UCI machine learning Data Repository.

http://archive.ics.uci.edu/ml/

$\endgroup$
1
24
votes
$\begingroup$

See my response to "Datasets for Running Statistical Analysis on" in reference to datasets in R.

$\endgroup$
1
  • $\begingroup$ you should copy that here before it gets deleted... $\endgroup$
    – naught101
    Commented Nov 21, 2012 at 7:34
19
votes
$\begingroup$

World Bank offers quite a lot of interesting data and has been recently very active in developing nice API for it.

Also, commugrate project has an interesting list available.

For US health related data head for Health Indicators Warehouse.

Daniel Lemire's blog points to few interesting examples (mostly tailored towards DB research) including Canadian Census 1880 and synoptic cloud reports.

And as for today (03/04/2012) US 1940 census records are also available to download.

$\endgroup$
1
  • 2
    $\begingroup$ The World Bank is going the extra mile with open data and maps, for Stata and R. $\endgroup$
    – Fr.
    Commented Mar 18, 2012 at 16:02
16
votes
$\begingroup$

Gapminder has a number (430 at the last look) of datasets, which may or may not be of use to you.

$\endgroup$
13
votes
$\begingroup$

MLComp has quite a few interesting datasets, and as a bonus your algorithm will get ranked if you upload it.

$\endgroup$
11
votes
$\begingroup$

A good place to look is Carnegie Mellon University's Data and Story Library or DASL, which contains data files that "illustrate the use of basic statistics methods... A good example can make a lesson on a particular statistics method vivid and relevant. DASL is designed to help teachers locate and identify datafiles for teaching. We hope that DASL will also serve as an archive for datasets from the statistics literature."

$\endgroup$
10
votes
$\begingroup$

Start R and type data(). This will show all datasets in the search path. Many additional datasets are available in add-on packages. For example, there are some interesting real-world social science datasets in the AER package.

$\endgroup$
9
votes
$\begingroup$

NIST provides a Reference Dataset archive.

$\endgroup$
7
votes
$\begingroup$

http://www.reddit.com/r/datasets and also, http://www.reddit.com/r/opendata both contain a constantly growing list of pointers to various datasets.

$\endgroup$
5
votes
$\begingroup$

The Stack Exchange network now has a new site, Open Data (in beta as of March 5th, 2015), dedicated to data. It describes itself as:

Open Data Stack Exchange is a question and answer site for developers and researchers interested in open data. It's built and run by you as part of the Stack Exchange network of Q&A sites. With your help, we're working together to build a library of detailed answers to every question about open data.

"Open data" refers to datasets that are "freely available to everyone to use and republish as they wish, without restrictions from copyright, patents or other mechanisms of control" (Wikipedia). However, the site seems amenable to requests for closed datasets.

$\endgroup$
4
votes
$\begingroup$

Timetric provides a web interface to data and provide a list of the publicly available data sets they use.

$\endgroup$
3
votes
$\begingroup$

Adding a couple to the list:

Both available via a REST API and have free trial plans.

$\endgroup$
2
votes
$\begingroup$

This is probably the most complete list you'll find: Some Datasets Available on the Web

$\endgroup$
2
votes
$\begingroup$

Peter Skomoroch maintains a list of datasets at http://www.datawrangling.com/some-datasets-available-on-the-web. Many of the links provided as to places that list datasets.

$\endgroup$
2
votes
$\begingroup$

I highly recommend checking out quandl.com. This is a data programmers dream. It provides one very easy API to access any of the over 10 million different data sits. You are looking for bi-modial or multi-variate data, so I would suggest checking out the various sets of population data eg this world population chart contains the sub component countries and territories that go into the total.

$\endgroup$
1
  • 1
    $\begingroup$ Some quandl data is free, some "Premium" i.e. costs $$. Also my API dream includes time-series nrows, ncols, and online plots (I want a pony). $\endgroup$
    – denis
    Commented Jun 16, 2015 at 12:52
2
votes
$\begingroup$

Data sets from seminal book A handbook of small data sets are available here.

$\endgroup$
2
votes
$\begingroup$

Here's another list that might be of help.

$\endgroup$
2
votes
$\begingroup$

Searching for an appropriate data set for my needs I have just stumbled across two sites that are pertinent to this discussion.

Datacite.org which describes itself as...

We are an international organisation which aims to:

  • establish easier access to research data
  • increase acceptance of research data as legitimate contributions in the scholarly record, and to
  • support data archiving to permit results to be verified and re-purposed for future study.

DataBib.org which describes itself as...

Databib is a tool for helping people identify and locate online repositories of research data. Users and bibliographers create and curate records that describe data repositories that users can search.

Thought it would be worth adding it to the list here for others.

Now to find something within its links that fits my needs!

$\endgroup$
1
vote
$\begingroup$

Usage Over Time

A very large Excel spreadsheet available for download containing data points for all online activities, with user demographics, over time. Please read Tip Sheet (below) before downloading or using this spreadsheet.

http://pewinternet.org/Trend-Data/Usage-Over-Time.aspx

$\endgroup$
1
vote
$\begingroup$

http://www.ckan.net has a number of datasets too.

http://www.biotorrents.net/browse.php is also starting to have quite a large amount of BIG datasets.

$\endgroup$
1
vote
$\begingroup$

Here's a list of quantitative social science data: https://f.briatte.org/teaching/quanti

I update it every 3-4 months or so.

At the very end, I cite the "Data is Plural" newsletter, which has tons of interesting stuff. It's my favourite newsletter ever.

$\endgroup$
1
vote
$\begingroup$

SODA POP at Penn State;

Simple Online Data Archive for POPulation studies.

$\endgroup$
0
votes
$\begingroup$

I'm gonna go ahead and bump an old topic because I just found this mother lode:

http://vincentarelbundock.github.io/Rdatasets/

$\endgroup$
0
votes
$\begingroup$

Singapore announces Open Data initiative. Check out data.gov.sg similar to data.gov in the US.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.